[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:21:12 dist-test-slave-920p krb5kdc[2279](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:21:12 dist-test-slave-920p krb5kdc[2279](info): set up 2 sockets
Sep 28 12:21:12 dist-test-slave-920p krb5kdc[2279](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 12:21:13.610733 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:21:13 dist-test-slave-920p krb5kdc[2279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:21:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:21:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
I20260928 12:21:13.741165 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:43607
--webserver_interface=127.2.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:40859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:43607
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:13.841075 2292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:13.841418 2292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:13.841457 2292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:13.842947 2292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:21:13.843004 2292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:13.843025 2292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:13.843042 2292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:21:13.843056 2292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:21:13.844798 2292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:40859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:43607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:43607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:13.845115 2292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:13.845415 2292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:13.851540 2300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:13.851739 2298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:13.851687 2297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:13.851783 2292 server_base.cc:1034] running on GCE node
I20260928 12:21:13.852272 2292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:13.852662 2292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:13.853883 2292 hybrid_clock.cc:648] HybridClock initialized: now 1790598073853880 us; error 84 us; skew 500 ppm
Sep 28 12:21:13 dist-test-slave-920p krb5kdc[2279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598073, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:13.857048 2292 init.cc:377] Logged in from keytab as kudu/127.2.56.62@KRBTEST.COM (short username kudu)
I20260928 12:21:13.857688 2292 webserver.cc:460] Webserver started at http://127.2.56.62:38483/ using document root <none> and password file <none>
I20260928 12:21:13.857936 2292 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:13.857990 2292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:13.858130 2292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:13.859501 2292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "b8614198cb514d79ba1bc29945bb2f4b"
format_stamp: "Formatted at 2026-09-28 12:21:13 on dist-test-slave-920p"
server_key: "af64ea554b4d38c298ea11b8329acaff"
server_key_iv: "ec404a4f4b702ca980caa6ebec458870"
server_key_version: "encryptionkey@0"
I20260928 12:21:13.859925 2292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "b8614198cb514d79ba1bc29945bb2f4b"
format_stamp: "Formatted at 2026-09-28 12:21:13 on dist-test-slave-920p"
server_key: "af64ea554b4d38c298ea11b8329acaff"
server_key_iv: "ec404a4f4b702ca980caa6ebec458870"
server_key_version: "encryptionkey@0"
I20260928 12:21:13.861485 2292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:21:13.862406 2307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:13.862623 2292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:21:13.862731 2292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "b8614198cb514d79ba1bc29945bb2f4b"
format_stamp: "Formatted at 2026-09-28 12:21:13 on dist-test-slave-920p"
server_key: "af64ea554b4d38c298ea11b8329acaff"
server_key_iv: "ec404a4f4b702ca980caa6ebec458870"
server_key_version: "encryptionkey@0"
I20260928 12:21:13.862869 2292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:13.880690 2292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:13.883658 2292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:13.883852 2292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:13.888813 2292 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:43607
I20260928 12:21:13.888855 2359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:43607 every 8 connection(s)
I20260928 12:21:13.889374 2292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:21:13.890633 2360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:13.893651 2360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8614198cb514d79ba1bc29945bb2f4b: Bootstrap starting.
I20260928 12:21:13.894387 2360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8614198cb514d79ba1bc29945bb2f4b: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:13.894672 2360 log.cc:824] T 00000000000000000000000000000000 P b8614198cb514d79ba1bc29945bb2f4b: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:13.895598 2360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8614198cb514d79ba1bc29945bb2f4b: No bootstrap required, opened a new log
I20260928 12:21:13.897650 2360 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8614198cb514d79ba1bc29945bb2f4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8614198cb514d79ba1bc29945bb2f4b" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43607 } }
I20260928 12:21:13.897852 2360 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8614198cb514d79ba1bc29945bb2f4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:13.897877 2360 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8614198cb514d79ba1bc29945bb2f4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8614198cb514d79ba1bc29945bb2f4b, State: Initialized, Role: FOLLOWER
I20260928 12:21:13.898064 2360 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8614198cb514d79ba1bc29945bb2f4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8614198cb514d79ba1bc29945bb2f4b" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43607 } }
I20260928 12:21:13.898164 2360 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8614198cb514d79ba1bc29945bb2f4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:21:13.898201 2360 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8614198cb514d79ba1bc29945bb2f4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:21:13.898253 2360 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8614198cb514d79ba1bc29945bb2f4b [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:13.898428 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 2292
I20260928 12:21:13.898558 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:21:13.899168 2360 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8614198cb514d79ba1bc29945bb2f4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8614198cb514d79ba1bc29945bb2f4b" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43607 } }
I20260928 12:21:13.899295 2360 leader_election.cc:302] T 00000000000000000000000000000000 P b8614198cb514d79ba1bc29945bb2f4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8614198cb514d79ba1bc29945bb2f4b; no voters:
I20260928 12:21:13.899427 2272 external_mini_cluster.cc:1428] Setting key 854ec07f616712e8b2c03b9218b0e0d5
I20260928 12:21:13.899523 2360 leader_election.cc:288] T 00000000000000000000000000000000 P b8614198cb514d79ba1bc29945bb2f4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:21:13.899607 2363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8614198cb514d79ba1bc29945bb2f4b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:21:13.899796 2363 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8614198cb514d79ba1bc29945bb2f4b [term 1 LEADER]: Becoming Leader. State: Replica: b8614198cb514d79ba1bc29945bb2f4b, State: Running, Role: LEADER
I20260928 12:21:13.899847 2360 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8614198cb514d79ba1bc29945bb2f4b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:21:13.899933 2363 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8614198cb514d79ba1bc29945bb2f4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8614198cb514d79ba1bc29945bb2f4b" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43607 } }
I20260928 12:21:13.900575 2364 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8614198cb514d79ba1bc29945bb2f4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8614198cb514d79ba1bc29945bb2f4b. Latest consensus state: current_term: 1 leader_uuid: "b8614198cb514d79ba1bc29945bb2f4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8614198cb514d79ba1bc29945bb2f4b" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43607 } } }
I20260928 12:21:13.900676 2364 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8614198cb514d79ba1bc29945bb2f4b [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:13.900904 2372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:21:13.901574 2365 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8614198cb514d79ba1bc29945bb2f4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8614198cb514d79ba1bc29945bb2f4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8614198cb514d79ba1bc29945bb2f4b" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43607 } } }
I20260928 12:21:13.901654 2365 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8614198cb514d79ba1bc29945bb2f4b [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:13.901710 2372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:21:13.903522 2372 catalog_manager.cc:1347] Generated new cluster ID: 7430b9520c3042bcab32128294519011
I20260928 12:21:13.903569 2372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:21:13.911999 2372 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:21:13.912663 2372 catalog_manager.cc:1504] Loading token signing keys...
Sep 28 12:21:13 dist-test-slave-920p krb5kdc[2279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:13.915807 2372 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8614198cb514d79ba1bc29945bb2f4b: Generated new TSK 0
I20260928 12:21:13.915982 2372 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:21:13.920759 2379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:13.901222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55554 (local address 127.2.56.62:43607)
0928 12:21:13.901536 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:13.901551 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:13.902099 (+ 548us) server_negotiation.cc:407] Connection header received
0928 12:21:13.904350 (+ 2251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:13.904358 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:13.904416 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:13.904572 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:13.906967 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:13.908521 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:13.910304 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:13.911003 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:13.911129 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:13.916507 (+ 5378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:13.916513 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:13.916513 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:13.916528 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:13.918519 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:13.919104 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:13.919107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:13.919108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:13.919221 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:13.919525 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:13.919526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:13.919526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:13.919788 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:13.919921 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:13.920308 (+ 387us) server_negotiation.cc:299] Negotiation successful
0928 12:21:13.920465 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":103,"threads_started":1}
WARNING: no policy specified for kudu/127.2.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab.
Entry for principal kudu/127.2.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab.
I20260928 12:21:13.948959 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.1:0
--local_ip_for_outbound_sockets=127.2.56.1
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:43607
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:40859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:14.042654 2386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:14.042912 2386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:14.042936 2386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:14.044173 2386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:14.044219 2386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:14.044251 2386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.1
I20260928 12:21:14.045806 2386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:40859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:43607
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.1
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:14.046082 2386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:14.046330 2386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:14.051632 2394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:14.051645 2392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:14.051645 2391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:14.051895 2386 server_base.cc:1034] running on GCE node
I20260928 12:21:14.052140 2386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:14.052398 2386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:14.053535 2386 hybrid_clock.cc:648] HybridClock initialized: now 1790598074053525 us; error 32 us; skew 500 ppm
Sep 28 12:21:14 dist-test-slave-920p krb5kdc[2279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598074, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:14.062896 2386 init.cc:377] Logged in from keytab as kudu/127.2.56.1@KRBTEST.COM (short username kudu)
I20260928 12:21:14.063473 2386 webserver.cc:460] Webserver started at http://127.2.56.1:46623/ using document root <none> and password file <none>
I20260928 12:21:14.063719 2386 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:14.063774 2386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:14.063920 2386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:14.065184 2386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-0/data/instance:
uuid: "400452ed823d41ae912d8293b61b0f77"
format_stamp: "Formatted at 2026-09-28 12:21:14 on dist-test-slave-920p"
server_key: "1ee67e233a528e8efd8634d798ae9b65"
server_key_iv: "b0d53dabea28537169c3c2696ee849ee"
server_key_version: "encryptionkey@0"
I20260928 12:21:14.065701 2386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance:
uuid: "400452ed823d41ae912d8293b61b0f77"
format_stamp: "Formatted at 2026-09-28 12:21:14 on dist-test-slave-920p"
server_key: "1ee67e233a528e8efd8634d798ae9b65"
server_key_iv: "b0d53dabea28537169c3c2696ee849ee"
server_key_version: "encryptionkey@0"
I20260928 12:21:14.067327 2386 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:21:14.068099 2401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:14.068260 2386 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:14.068349 2386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-0/wal
uuid: "400452ed823d41ae912d8293b61b0f77"
format_stamp: "Formatted at 2026-09-28 12:21:14 on dist-test-slave-920p"
server_key: "1ee67e233a528e8efd8634d798ae9b65"
server_key_iv: "b0d53dabea28537169c3c2696ee849ee"
server_key_version: "encryptionkey@0"
I20260928 12:21:14.068434 2386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:14.086472 2386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:14.089411 2386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:14.089591 2386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:14.089910 2386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:21:14.090358 2386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:14.090406 2386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:14.090451 2386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:14.090474 2386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:14.098315 2386 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.1:33575
I20260928 12:21:14.098387 2514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.1:33575 every 8 connection(s)
I20260928 12:21:14.098838 2386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
I20260928 12:21:14.105468 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 2386
I20260928 12:21:14.105576 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance
I20260928 12:21:14.105798 2272 external_mini_cluster.cc:1428] Setting key 34cc54091078a4a4d7ac1efdb284b14f
Sep 28 12:21:14 dist-test-slave-920p krb5kdc[2279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598074, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:14.112294 2379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:14.099889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:54233 (local address 127.2.56.62:43607)
0928 12:21:14.099955 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:14.099958 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:14.100185 (+ 227us) server_negotiation.cc:407] Connection header received
0928 12:21:14.100975 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:14.100977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:14.100995 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:14.101055 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:14.102558 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:14.103162 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:14.104408 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:14.104653 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:14.104719 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:14.108222 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:14.108225 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:14.108225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:14.108237 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:14.110138 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:14.110785 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:14.110786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:14.110787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:14.110872 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:14.111267 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:14.111269 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:14.111269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:14.111444 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:14.111527 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:14.112037 (+ 510us) server_negotiation.cc:299] Negotiation successful
0928 12:21:14.112182 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 12:21:14.112617 2517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:14.099804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43607 (local address 127.2.56.1:54233)
0928 12:21:14.100020 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:14.100076 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:14.100815 (+ 739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:14.101164 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:14.101170 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:14.102123 (+ 953us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:14.102430 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:14.102438 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:14.103271 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:14.103273 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:14.104300 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:14.104302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:14.104740 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:14.104741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:14.104935 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:14.105736 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:14.105751 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:14.108077 (+ 2326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:14.110247 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:14.110261 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:14.110266 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:14.110666 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:14.110951 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:14.110952 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:14.110953 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:14.111151 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:14.111534 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:14.111544 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:14.111833 (+ 289us) client_negotiation.cc:769] Sending connection context
0928 12:21:14.111991 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 12:21:14.112277 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":100,"threads_started":1}
I20260928 12:21:14.113490 2515 heartbeater.cc:347] Connected to a master server at 127.2.56.62:43607
I20260928 12:21:14.113656 2515 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:14.113986 2515 heartbeater.cc:511] Master 127.2.56.62:43607 requested a full tablet report, sending...
I20260928 12:21:14.114939 2324 ts_manager.cc:194] Registered new tserver with Master: 400452ed823d41ae912d8293b61b0f77 (127.2.56.1:33575)
I20260928 12:21:14.115710 2324 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.1@KRBTEST.COM'} at 127.2.56.1:54233
WARNING: no policy specified for kudu/127.2.56.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab.
Entry for principal kudu/127.2.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab.
I20260928 12:21:14.137108 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.2:0
--local_ip_for_outbound_sockets=127.2.56.2
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:43607
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:40859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:14.235848 2520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:14.236135 2520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:14.236164 2520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:14.237637 2520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:14.237686 2520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:14.237720 2520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.2
I20260928 12:21:14.239530 2520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:40859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:43607
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.2
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:14.239810 2520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:14.240075 2520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:14.245373 2526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:14.245430 2525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:14.245440 2520 server_base.cc:1034] running on GCE node
W20260928 12:21:14.245404 2528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:14.245775 2520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:14.246021 2520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:14.247195 2520 hybrid_clock.cc:648] HybridClock initialized: now 1790598074247183 us; error 29 us; skew 500 ppm
Sep 28 12:21:14 dist-test-slave-920p krb5kdc[2279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598074, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:14.250146 2520 init.cc:377] Logged in from keytab as kudu/127.2.56.2@KRBTEST.COM (short username kudu)
I20260928 12:21:14.250617 2520 webserver.cc:460] Webserver started at http://127.2.56.2:33041/ using document root <none> and password file <none>
I20260928 12:21:14.250844 2520 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:14.250893 2520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:14.251035 2520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:14.252200 2520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-1/data/instance:
uuid: "8f9e34ce3aa745a6af3d2ccdf773efc8"
format_stamp: "Formatted at 2026-09-28 12:21:14 on dist-test-slave-920p"
server_key: "7de6a0abc7acaaad7959b63a90d5b045"
server_key_iv: "05d8eacf5e2c5d89e2ab866b973dc5fa"
server_key_version: "encryptionkey@0"
I20260928 12:21:14.252568 2520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance:
uuid: "8f9e34ce3aa745a6af3d2ccdf773efc8"
format_stamp: "Formatted at 2026-09-28 12:21:14 on dist-test-slave-920p"
server_key: "7de6a0abc7acaaad7959b63a90d5b045"
server_key_iv: "05d8eacf5e2c5d89e2ab866b973dc5fa"
server_key_version: "encryptionkey@0"
I20260928 12:21:14.253937 2520 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:21:14.254657 2535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:14.254869 2520 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:14.254962 2520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-1/wal
uuid: "8f9e34ce3aa745a6af3d2ccdf773efc8"
format_stamp: "Formatted at 2026-09-28 12:21:14 on dist-test-slave-920p"
server_key: "7de6a0abc7acaaad7959b63a90d5b045"
server_key_iv: "05d8eacf5e2c5d89e2ab866b973dc5fa"
server_key_version: "encryptionkey@0"
I20260928 12:21:14.255020 2520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:14.266603 2520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:14.268499 2520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:14.268635 2520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:14.268862 2520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:21:14.269177 2520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:14.269209 2520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:14.269239 2520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:14.269306 2520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:14.275549 2520 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.2:45071
I20260928 12:21:14.275621 2648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.2:45071 every 8 connection(s)
I20260928 12:21:14.275951 2520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
I20260928 12:21:14.282817 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 2520
I20260928 12:21:14.282938 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance
I20260928 12:21:14.283118 2272 external_mini_cluster.cc:1428] Setting key 57cc8a81ed86808753739c10baff9a6f
Sep 28 12:21:14 dist-test-slave-920p krb5kdc[2279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598074, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:14.287561 2379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:14.276682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:56375 (local address 127.2.56.62:43607)
0928 12:21:14.276732 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:14.276734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:14.276973 (+ 239us) server_negotiation.cc:407] Connection header received
0928 12:21:14.277672 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:14.277674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:14.277689 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:14.277765 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:14.279027 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:14.279614 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:14.280870 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:14.281119 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:14.281188 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:14.283931 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:14.283933 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:14.283934 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:14.283943 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:14.285521 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:14.286154 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:14.286156 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:14.286156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:14.286227 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:14.286712 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:14.286714 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:14.286714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:14.286880 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:14.286949 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:14.287324 (+ 375us) server_negotiation.cc:299] Negotiation successful
0928 12:21:14.287449 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:21:14.287775 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:14.276670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43607 (local address 127.2.56.2:56375)
0928 12:21:14.276832 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:14.276873 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:14.277533 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:14.277955 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:14.277960 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:14.278626 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:14.278896 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:14.278903 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:14.279735 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:14.279737 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:14.280694 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:14.280696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:14.281223 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:14.281224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:14.281438 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:14.282060 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:14.282070 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:14.283813 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:14.285624 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:14.285634 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:14.285638 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:14.286035 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:14.286303 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:14.286303 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:14.286304 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:14.286570 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:14.286952 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:14.286960 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:14.287145 (+ 185us) client_negotiation.cc:769] Sending connection context
0928 12:21:14.287269 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 12:21:14.287497 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
I20260928 12:21:14.288352 2649 heartbeater.cc:347] Connected to a master server at 127.2.56.62:43607
I20260928 12:21:14.288467 2649 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:14.288698 2649 heartbeater.cc:511] Master 127.2.56.62:43607 requested a full tablet report, sending...
I20260928 12:21:14.289211 2324 ts_manager.cc:194] Registered new tserver with Master: 8f9e34ce3aa745a6af3d2ccdf773efc8 (127.2.56.2:45071)
I20260928 12:21:14.289674 2324 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.2@KRBTEST.COM'} at 127.2.56.2:56375
WARNING: no policy specified for kudu/127.2.56.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab.
Entry for principal kudu/127.2.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab.
I20260928 12:21:14.310237 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.3:0
--local_ip_for_outbound_sockets=127.2.56.3
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:43607
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:40859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:14.397960 2654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:14.398203 2654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:14.398236 2654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:14.399498 2654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:14.399544 2654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:14.399580 2654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.3
I20260928 12:21:14.401342 2654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:40859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:43607
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.3
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:14.401608 2654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:14.401892 2654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:14.407181 2662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:14.407181 2659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:14.407267 2654 server_base.cc:1034] running on GCE node
W20260928 12:21:14.407207 2660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:14.407606 2654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:14.407857 2654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:14.409001 2654 hybrid_clock.cc:648] HybridClock initialized: now 1790598074408985 us; error 33 us; skew 500 ppm
Sep 28 12:21:14 dist-test-slave-920p krb5kdc[2279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598074, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:14.411657 2654 init.cc:377] Logged in from keytab as kudu/127.2.56.3@KRBTEST.COM (short username kudu)
I20260928 12:21:14.412127 2654 webserver.cc:460] Webserver started at http://127.2.56.3:38997/ using document root <none> and password file <none>
I20260928 12:21:14.412343 2654 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:14.412391 2654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:14.412533 2654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:14.413826 2654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-2/data/instance:
uuid: "1b23e8ef67a04cbd95550364026a6477"
format_stamp: "Formatted at 2026-09-28 12:21:14 on dist-test-slave-920p"
server_key: "78969ca181f121fe7afe1b441419bff7"
server_key_iv: "f37f9e4d405cacb26267520016b965fe"
server_key_version: "encryptionkey@0"
I20260928 12:21:14.414193 2654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance:
uuid: "1b23e8ef67a04cbd95550364026a6477"
format_stamp: "Formatted at 2026-09-28 12:21:14 on dist-test-slave-920p"
server_key: "78969ca181f121fe7afe1b441419bff7"
server_key_iv: "f37f9e4d405cacb26267520016b965fe"
server_key_version: "encryptionkey@0"
I20260928 12:21:14.415566 2654 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:21:14.416345 2669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:14.416491 2654 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:14.416579 2654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-2/wal
uuid: "1b23e8ef67a04cbd95550364026a6477"
format_stamp: "Formatted at 2026-09-28 12:21:14 on dist-test-slave-920p"
server_key: "78969ca181f121fe7afe1b441419bff7"
server_key_iv: "f37f9e4d405cacb26267520016b965fe"
server_key_version: "encryptionkey@0"
I20260928 12:21:14.416646 2654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:14.433691 2654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:14.435777 2654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:14.435920 2654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:14.436190 2654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:21:14.436544 2654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:14.436595 2654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:14.436627 2654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:14.436648 2654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:14.442866 2654 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.3:32885
I20260928 12:21:14.442920 2782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.3:32885 every 8 connection(s)
I20260928 12:21:14.443274 2654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
I20260928 12:21:14.445309 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 2654
I20260928 12:21:14.445397 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance
I20260928 12:21:14.445580 2272 external_mini_cluster.cc:1428] Setting key 52bcb68babdb0bd450d4316e3e3395dd
Sep 28 12:21:14 dist-test-slave-920p krb5kdc[2279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598074, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:14.453038 2379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:14.444115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:56151 (local address 127.2.56.62:43607)
0928 12:21:14.444182 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:14.444184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:14.444408 (+ 224us) server_negotiation.cc:407] Connection header received
0928 12:21:14.444995 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:14.444997 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:14.445012 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:14.445060 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:14.446271 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:14.446855 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:14.447948 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:14.448153 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:14.448212 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:14.450515 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:14.450518 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:14.450518 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:14.450527 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:14.451733 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:14.452114 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:14.452115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:14.452115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:14.452167 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:14.452385 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:14.452386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:14.452386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:14.452512 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:14.452569 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:14.452847 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 12:21:14.452931 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 12:21:14.453308 2785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:14.444089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43607 (local address 127.2.56.3:56151)
0928 12:21:14.444278 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:14.444319 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:14.444887 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:14.445173 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:14.445178 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:14.445895 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:14.446184 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:14.446193 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:14.446939 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:14.446940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:14.447866 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:14.447869 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:14.448222 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:14.448223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:14.448368 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:14.448866 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:14.448875 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:14.450417 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:14.451803 (+ 1386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:14.451811 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:14.451813 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:14.452048 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:14.452218 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:14.452219 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:14.452220 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:14.452330 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:14.452565 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:14.452571 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:14.452724 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 12:21:14.452817 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 12:21:14.453023 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":107,"threads_started":1}
I20260928 12:21:14.453795 2783 heartbeater.cc:347] Connected to a master server at 127.2.56.62:43607
I20260928 12:21:14.453894 2783 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:14.454085 2783 heartbeater.cc:511] Master 127.2.56.62:43607 requested a full tablet report, sending...
I20260928 12:21:14.454455 2324 ts_manager.cc:194] Registered new tserver with Master: 1b23e8ef67a04cbd95550364026a6477 (127.2.56.3:32885)
I20260928 12:21:14.454831 2324 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.3@KRBTEST.COM'} at 127.2.56.3:56151
I20260928 12:21:14.457741 2272 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 12:21:14 dist-test-slave-920p krb5kdc[2279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598074, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 12:21:14 dist-test-slave-920p krb5kdc[2279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598074, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.56.1@KRBTEST.COM
I20260928 12:21:14.482024 2790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:14.474211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56036 (local address 127.2.56.1:33575)
0928 12:21:14.474398 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:14.474405 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:14.474417 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:21:14.474450 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:14.474453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:14.474470 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:14.474544 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:14.475127 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:14.475951 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:14.476722 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:14.477138 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:14.477198 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:14.479548 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:14.479559 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:14.479559 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:14.479568 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:14.480991 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:14.481337 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:14.481341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:14.481341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:14.481394 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:14.481530 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:14.481530 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:14.481531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:14.481674 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:14.481788 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:14.481821 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 12:21:14.481885 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":91,"threads_started":1}
W20260928 12:21:14.482430 2428 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56036
Sep 28 12:21:14 dist-test-slave-920p krb5kdc[2279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 12:21:14 dist-test-slave-920p krb5kdc[2279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.1@KRBTEST.COM
I20260928 12:21:14.504458 2790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:14.497763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56042 (local address 127.2.56.1:33575)
0928 12:21:14.497813 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:14.497815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:14.497871 (+ 56us) server_negotiation.cc:407] Connection header received
0928 12:21:14.497952 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:14.497952 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:14.497984 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:14.498032 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:14.498572 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:14.499115 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:14.499859 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:14.500092 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:14.500144 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:14.502388 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:14.502390 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:14.502390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:14.502397 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:14.503522 (+ 1125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:14.503800 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:14.503801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:14.503801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:14.503843 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:14.503992 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:14.503993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:14.503993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:14.504083 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:14.504174 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:14.504315 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 12:21:14.504378 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:21:14.505095 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 2386
I20260928 12:21:14.510488 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 2520
I20260928 12:21:14.515839 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 2654
I20260928 12:21:14.519912 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 2292
2026-09-28T12:21:14Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1885 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:21:14 dist-test-slave-920p krb5kdc[2797](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:21:14 dist-test-slave-920p krb5kdc[2797](info): set up 2 sockets
Sep 28 12:21:14 dist-test-slave-920p krb5kdc[2797](info): commencing operation
krb5kdc: starting...
W20260928 12:21:15.473053 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:21:15 dist-test-slave-920p krb5kdc[2797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:21:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:21:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
I20260928 12:21:15.592098 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:36579
--webserver_interface=127.2.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:39891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:36579
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:15.682067 2810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:15.682312 2810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:15.682333 2810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:15.683619 2810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:21:15.683672 2810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:15.683686 2810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:15.683696 2810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:21:15.683708 2810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:21:15.685330 2810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:39891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:36579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:36579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:15.685613 2810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:15.685894 2810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:15.691080 2816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:15.691246 2810 server_base.cc:1034] running on GCE node
W20260928 12:21:15.691114 2818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:15.691107 2815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:15.691607 2810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:15.691908 2810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:15.693043 2810 hybrid_clock.cc:648] HybridClock initialized: now 1790598075693034 us; error 29 us; skew 500 ppm
Sep 28 12:21:15 dist-test-slave-920p krb5kdc[2797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598075, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:15.696089 2810 init.cc:377] Logged in from keytab as kudu/127.2.56.62@KRBTEST.COM (short username kudu)
I20260928 12:21:15.696600 2810 webserver.cc:460] Webserver started at http://127.2.56.62:46707/ using document root <none> and password file <none>
I20260928 12:21:15.696889 2810 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:15.696941 2810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:15.697072 2810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:15.698383 2810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "5135bd2c2dda494989962d6f4acce1c9"
format_stamp: "Formatted at 2026-09-28 12:21:15 on dist-test-slave-920p"
server_key: "04bf0e128fafaab5b1fda63782c3b4ad"
server_key_iv: "920e6456648a028e7fc2f7905a67b2a3"
server_key_version: "encryptionkey@0"
I20260928 12:21:15.698761 2810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "5135bd2c2dda494989962d6f4acce1c9"
format_stamp: "Formatted at 2026-09-28 12:21:15 on dist-test-slave-920p"
server_key: "04bf0e128fafaab5b1fda63782c3b4ad"
server_key_iv: "920e6456648a028e7fc2f7905a67b2a3"
server_key_version: "encryptionkey@0"
I20260928 12:21:15.700268 2810 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 12:21:15.700964 2825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:15.701159 2810 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:15.701252 2810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "5135bd2c2dda494989962d6f4acce1c9"
format_stamp: "Formatted at 2026-09-28 12:21:15 on dist-test-slave-920p"
server_key: "04bf0e128fafaab5b1fda63782c3b4ad"
server_key_iv: "920e6456648a028e7fc2f7905a67b2a3"
server_key_version: "encryptionkey@0"
I20260928 12:21:15.701354 2810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:15.718079 2810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:15.720466 2810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:15.720590 2810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:15.725055 2810 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:36579
I20260928 12:21:15.725099 2877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:36579 every 8 connection(s)
I20260928 12:21:15.725512 2810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:21:15.726596 2878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:15.727598 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 2810
I20260928 12:21:15.727679 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:21:15.727833 2272 external_mini_cluster.cc:1428] Setting key 2e952438a585809f9bd78c1da8e99e87
I20260928 12:21:15.730072 2878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5135bd2c2dda494989962d6f4acce1c9: Bootstrap starting.
I20260928 12:21:15.730770 2878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5135bd2c2dda494989962d6f4acce1c9: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:15.731087 2878 log.cc:824] T 00000000000000000000000000000000 P 5135bd2c2dda494989962d6f4acce1c9: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:15.731781 2878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5135bd2c2dda494989962d6f4acce1c9: No bootstrap required, opened a new log
I20260928 12:21:15.733891 2878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5135bd2c2dda494989962d6f4acce1c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5135bd2c2dda494989962d6f4acce1c9" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 36579 } }
I20260928 12:21:15.734063 2878 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5135bd2c2dda494989962d6f4acce1c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:15.734098 2878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5135bd2c2dda494989962d6f4acce1c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5135bd2c2dda494989962d6f4acce1c9, State: Initialized, Role: FOLLOWER
I20260928 12:21:15.734241 2878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5135bd2c2dda494989962d6f4acce1c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5135bd2c2dda494989962d6f4acce1c9" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 36579 } }
I20260928 12:21:15.734318 2878 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5135bd2c2dda494989962d6f4acce1c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:21:15.734357 2878 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5135bd2c2dda494989962d6f4acce1c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:21:15.734408 2878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5135bd2c2dda494989962d6f4acce1c9 [term 0 FOLLOWER]: Advancing to term 1
Sep 28 12:21:15 dist-test-slave-920p krb5kdc[2797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:15.735025 2878 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5135bd2c2dda494989962d6f4acce1c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5135bd2c2dda494989962d6f4acce1c9" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 36579 } }
I20260928 12:21:15.735131 2878 leader_election.cc:302] T 00000000000000000000000000000000 P 5135bd2c2dda494989962d6f4acce1c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5135bd2c2dda494989962d6f4acce1c9; no voters:
I20260928 12:21:15.735312 2878 leader_election.cc:288] T 00000000000000000000000000000000 P 5135bd2c2dda494989962d6f4acce1c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:21:15.735351 2883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5135bd2c2dda494989962d6f4acce1c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:21:15.735488 2883 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5135bd2c2dda494989962d6f4acce1c9 [term 1 LEADER]: Becoming Leader. State: Replica: 5135bd2c2dda494989962d6f4acce1c9, State: Running, Role: LEADER
I20260928 12:21:15.735595 2883 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5135bd2c2dda494989962d6f4acce1c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5135bd2c2dda494989962d6f4acce1c9" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 36579 } }
I20260928 12:21:15.735632 2878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5135bd2c2dda494989962d6f4acce1c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:21:15.735971 2883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5135bd2c2dda494989962d6f4acce1c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5135bd2c2dda494989962d6f4acce1c9. Latest consensus state: current_term: 1 leader_uuid: "5135bd2c2dda494989962d6f4acce1c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5135bd2c2dda494989962d6f4acce1c9" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 36579 } } }
I20260928 12:21:15.736043 2883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5135bd2c2dda494989962d6f4acce1c9 [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:15.736195 2885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5135bd2c2dda494989962d6f4acce1c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5135bd2c2dda494989962d6f4acce1c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5135bd2c2dda494989962d6f4acce1c9" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 36579 } } }
I20260928 12:21:15.736249 2885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5135bd2c2dda494989962d6f4acce1c9 [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:15.736367 2887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:21:15.736937 2887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:21:15.738837 2887 catalog_manager.cc:1347] Generated new cluster ID: 327622d5a5014a4083b2ac624396a821
I20260928 12:21:15.738880 2887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:21:15.741567 2881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:15.728452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37876 (local address 127.2.56.62:36579)
0928 12:21:15.728657 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:15.728666 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:15.728695 (+ 29us) server_negotiation.cc:407] Connection header received
0928 12:21:15.729202 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:15.729211 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:15.729301 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:15.729476 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:15.730150 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:15.731239 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:15.732026 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:15.732478 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:15.732575 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:15.735117 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:15.735122 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:15.735123 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:15.735138 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:15.737249 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:15.737775 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:15.737779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:15.737780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:15.737872 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:15.738130 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:15.738131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:15.738131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:15.738422 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:15.738581 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:15.738711 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 12:21:15.738858 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":97,"threads_started":1}
I20260928 12:21:15.752149 2887 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:21:15.752980 2887 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:21:15.769322 2887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5135bd2c2dda494989962d6f4acce1c9: Generated new TSK 0
I20260928 12:21:15.769590 2887 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab.
Entry for principal kudu/127.2.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab.
I20260928 12:21:15.800328 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.1:0
--local_ip_for_outbound_sockets=127.2.56.1
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:36579
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:39891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:15.888564 2904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:15.888779 2904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:15.888803 2904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:15.890055 2904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:15.890101 2904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:15.890127 2904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.1
I20260928 12:21:15.891673 2904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:39891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:36579
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.1
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:15.891928 2904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:15.892189 2904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:21:15.897795 2904 server_base.cc:1034] running on GCE node
W20260928 12:21:15.897763 2910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:15.897763 2912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:15.897763 2909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:15.898237 2904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:15.898644 2904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:15.899807 2904 hybrid_clock.cc:648] HybridClock initialized: now 1790598075899793 us; error 39 us; skew 500 ppm
Sep 28 12:21:15 dist-test-slave-920p krb5kdc[2797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598075, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:15.902583 2904 init.cc:377] Logged in from keytab as kudu/127.2.56.1@KRBTEST.COM (short username kudu)
I20260928 12:21:15.903070 2904 webserver.cc:460] Webserver started at http://127.2.56.1:46187/ using document root <none> and password file <none>
I20260928 12:21:15.903272 2904 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:15.903313 2904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:15.903412 2904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:15.904637 2904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-0/data/instance:
uuid: "c51124ccc90547b5a4bfb9a33ed892d4"
format_stamp: "Formatted at 2026-09-28 12:21:15 on dist-test-slave-920p"
server_key: "b1e1aad4062ad9fef2d4fcbe6c853ec7"
server_key_iv: "91dba0069a2bf22c37afcdaf9445abda"
server_key_version: "encryptionkey@0"
I20260928 12:21:15.905045 2904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance:
uuid: "c51124ccc90547b5a4bfb9a33ed892d4"
format_stamp: "Formatted at 2026-09-28 12:21:15 on dist-test-slave-920p"
server_key: "b1e1aad4062ad9fef2d4fcbe6c853ec7"
server_key_iv: "91dba0069a2bf22c37afcdaf9445abda"
server_key_version: "encryptionkey@0"
I20260928 12:21:15.906419 2904 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:21:15.907137 2919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:15.907305 2904 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:15.907387 2904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-0/wal
uuid: "c51124ccc90547b5a4bfb9a33ed892d4"
format_stamp: "Formatted at 2026-09-28 12:21:15 on dist-test-slave-920p"
server_key: "b1e1aad4062ad9fef2d4fcbe6c853ec7"
server_key_iv: "91dba0069a2bf22c37afcdaf9445abda"
server_key_version: "encryptionkey@0"
I20260928 12:21:15.907442 2904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:15.915637 2904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:15.917536 2904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:15.917656 2904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:15.917867 2904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:21:15.918177 2904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:15.918207 2904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:15.918231 2904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:15.918246 2904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:15.924659 2904 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.1:35969
I20260928 12:21:15.924731 3032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.1:35969 every 8 connection(s)
I20260928 12:21:15.925063 2904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
I20260928 12:21:15.925532 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 2904
I20260928 12:21:15.925625 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance
I20260928 12:21:15.925818 2272 external_mini_cluster.cc:1428] Setting key 9bcb80fe2c00f3d4d8fed69446af14ed
Sep 28 12:21:15 dist-test-slave-920p krb5kdc[2797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598075, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.56.2@KRBTEST.COM; defaulting to no policy
I20260928 12:21:15.935827 2881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:15.925953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:53739 (local address 127.2.56.62:36579)
0928 12:21:15.926013 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:15.926015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:15.926275 (+ 260us) server_negotiation.cc:407] Connection header received
0928 12:21:15.927627 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:15.927628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:15.927639 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:15.927681 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:15.928597 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:15.929145 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:15.930302 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:15.930658 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:15.930742 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:15.933129 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:15.933130 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:15.933131 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:15.933139 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:15.934501 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:15.934850 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:15.934851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:15.934851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:15.934898 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:15.935154 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:15.935155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:15.935156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:15.935279 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:15.935342 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:15.935631 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 12:21:15.935721 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 12:21:15.936021 3035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:15.925953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:36579 (local address 127.2.56.1:53739)
0928 12:21:15.926128 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:15.926181 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:15.927508 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:15.927777 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:15.927781 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:15.928192 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:15.928497 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:15.928505 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:15.929231 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:15.929232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:15.930199 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:15.930201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:15.930742 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:15.930743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:15.930862 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:15.931421 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:15.931431 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:15.933029 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:15.934570 (+ 1541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:15.934577 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:15.934579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:15.934792 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:15.934947 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:15.934947 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:15.934948 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:15.935077 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:15.935338 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:15.935345 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:15.935505 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 12:21:15.935598 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 12:21:15.935768 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.2@KRBTEST.COM" created.
I20260928 12:21:15.936574 3033 heartbeater.cc:347] Connected to a master server at 127.2.56.62:36579
I20260928 12:21:15.936694 3033 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:15.936978 3033 heartbeater.cc:511] Master 127.2.56.62:36579 requested a full tablet report, sending...
I20260928 12:21:15.937717 2842 ts_manager.cc:194] Registered new tserver with Master: c51124ccc90547b5a4bfb9a33ed892d4 (127.2.56.1:35969)
I20260928 12:21:15.938365 2842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.1@KRBTEST.COM'} at 127.2.56.1:53739
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab.
Entry for principal kudu/127.2.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab.
I20260928 12:21:15.951542 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.2:0
--local_ip_for_outbound_sockets=127.2.56.2
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:36579
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:39891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:16.038141 3038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:16.038364 3038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:16.038388 3038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:16.039517 3038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:16.039559 3038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:16.039588 3038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.2
I20260928 12:21:16.041185 3038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:39891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:36579
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.2
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:16.041435 3038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:16.041639 3038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:16.046634 3046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:16.046691 3044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:16.046634 3043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:16.046872 3038 server_base.cc:1034] running on GCE node
I20260928 12:21:16.047112 3038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:16.047355 3038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:16.048501 3038 hybrid_clock.cc:648] HybridClock initialized: now 1790598076048486 us; error 35 us; skew 500 ppm
Sep 28 12:21:16 dist-test-slave-920p krb5kdc[2797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598076, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:16.051051 3038 init.cc:377] Logged in from keytab as kudu/127.2.56.2@KRBTEST.COM (short username kudu)
I20260928 12:21:16.051539 3038 webserver.cc:460] Webserver started at http://127.2.56.2:34285/ using document root <none> and password file <none>
I20260928 12:21:16.051752 3038 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:16.051798 3038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:16.051927 3038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:16.053030 3038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-1/data/instance:
uuid: "a358fff20a4d4d31935db08d80fd3409"
format_stamp: "Formatted at 2026-09-28 12:21:16 on dist-test-slave-920p"
server_key: "c76d4fe04b7d1263d532390d4a6662cf"
server_key_iv: "5851604dafe88589222c4e0cc5dd094a"
server_key_version: "encryptionkey@0"
I20260928 12:21:16.053406 3038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance:
uuid: "a358fff20a4d4d31935db08d80fd3409"
format_stamp: "Formatted at 2026-09-28 12:21:16 on dist-test-slave-920p"
server_key: "c76d4fe04b7d1263d532390d4a6662cf"
server_key_iv: "5851604dafe88589222c4e0cc5dd094a"
server_key_version: "encryptionkey@0"
I20260928 12:21:16.054695 3038 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:21:16.055348 3053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:16.055502 3038 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:21:16.055583 3038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-1/wal
uuid: "a358fff20a4d4d31935db08d80fd3409"
format_stamp: "Formatted at 2026-09-28 12:21:16 on dist-test-slave-920p"
server_key: "c76d4fe04b7d1263d532390d4a6662cf"
server_key_iv: "5851604dafe88589222c4e0cc5dd094a"
server_key_version: "encryptionkey@0"
I20260928 12:21:16.055646 3038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:16.068370 3038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:16.070382 3038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:16.070519 3038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:16.070744 3038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:21:16.071050 3038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:16.071085 3038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:16.071113 3038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:16.071137 3038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:16.077575 3038 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.2:43627
I20260928 12:21:16.077626 3166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.2:43627 every 8 connection(s)
I20260928 12:21:16.077968 3038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
Sep 28 12:21:16 dist-test-slave-920p krb5kdc[2797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598076, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:16.086557 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 3038
I20260928 12:21:16.086652 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance
I20260928 12:21:16.086844 2272 external_mini_cluster.cc:1428] Setting key ed4765ca61573849ff181327604c48e5
I20260928 12:21:16.087154 2881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:16.078699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:41809 (local address 127.2.56.62:36579)
0928 12:21:16.078761 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:16.078763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:16.078991 (+ 228us) server_negotiation.cc:407] Connection header received
0928 12:21:16.079988 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:16.079989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:16.079999 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:16.080055 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:16.080708 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:16.081239 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:16.082300 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:16.082500 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:16.082545 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:16.084634 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:16.084636 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:16.084636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:16.084641 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:16.085825 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:16.086191 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:16.086191 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:16.086192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:16.086235 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:16.086444 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:16.086444 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:16.086444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:16.086562 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:16.086607 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:16.086961 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 12:21:16.087052 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 12:21:16.087339 3169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:16.078681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:36579 (local address 127.2.56.2:41809)
0928 12:21:16.078838 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:16.078883 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:16.079900 (+ 1017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:16.080139 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:16.080142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:16.080421 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:16.080643 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:16.080648 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:16.081348 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:16.081349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:16.082235 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:16.082237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:16.082555 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:16.082556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:16.082657 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:16.083175 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:16.083185 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:16.084547 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:16.085909 (+ 1362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:16.085915 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:16.085917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:16.086137 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:16.086277 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:16.086277 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:16.086277 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:16.086393 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:16.086614 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:16.086621 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:16.086794 (+ 173us) client_negotiation.cc:769] Sending connection context
0928 12:21:16.086940 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 12:21:16.087099 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
I20260928 12:21:16.087764 3167 heartbeater.cc:347] Connected to a master server at 127.2.56.62:36579
I20260928 12:21:16.087870 3167 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:16.088078 3167 heartbeater.cc:511] Master 127.2.56.62:36579 requested a full tablet report, sending...
I20260928 12:21:16.088481 2842 ts_manager.cc:194] Registered new tserver with Master: a358fff20a4d4d31935db08d80fd3409 (127.2.56.2:43627)
I20260928 12:21:16.088858 2842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.2@KRBTEST.COM'} at 127.2.56.2:41809
WARNING: no policy specified for kudu/127.2.56.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab.
Entry for principal kudu/127.2.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab.
I20260928 12:21:16.111470 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.3:0
--local_ip_for_outbound_sockets=127.2.56.3
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:36579
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:39891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:16.202996 3172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:16.203344 3172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:16.203378 3172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:16.204988 3172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:16.205057 3172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:16.205096 3172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.3
I20260928 12:21:16.207190 3172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:39891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:36579
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.3
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:16.207499 3172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:16.207757 3172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:16.214112 3178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:16.214112 3177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:16.214115 3180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:16.214505 3172 server_base.cc:1034] running on GCE node
I20260928 12:21:16.214795 3172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:16.215118 3172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:16.216300 3172 hybrid_clock.cc:648] HybridClock initialized: now 1790598076216275 us; error 36 us; skew 500 ppm
Sep 28 12:21:16 dist-test-slave-920p krb5kdc[2797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598076, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:16.219764 3172 init.cc:377] Logged in from keytab as kudu/127.2.56.3@KRBTEST.COM (short username kudu)
I20260928 12:21:16.220572 3172 webserver.cc:460] Webserver started at http://127.2.56.3:42007/ using document root <none> and password file <none>
I20260928 12:21:16.220866 3172 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:16.220925 3172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:16.221055 3172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:16.222564 3172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-2/data/instance:
uuid: "9f223427b1ca45a2b9d4557837d0570e"
format_stamp: "Formatted at 2026-09-28 12:21:16 on dist-test-slave-920p"
server_key: "6be596c71ad9fa568c1a9a6aa8807393"
server_key_iv: "cc5220e835d3a26a0eb4e431b642f127"
server_key_version: "encryptionkey@0"
I20260928 12:21:16.222975 3172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance:
uuid: "9f223427b1ca45a2b9d4557837d0570e"
format_stamp: "Formatted at 2026-09-28 12:21:16 on dist-test-slave-920p"
server_key: "6be596c71ad9fa568c1a9a6aa8807393"
server_key_iv: "cc5220e835d3a26a0eb4e431b642f127"
server_key_version: "encryptionkey@0"
I20260928 12:21:16.224619 3172 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:21:16.225564 3187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:16.225847 3172 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:21:16.225957 3172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-2/wal
uuid: "9f223427b1ca45a2b9d4557837d0570e"
format_stamp: "Formatted at 2026-09-28 12:21:16 on dist-test-slave-920p"
server_key: "6be596c71ad9fa568c1a9a6aa8807393"
server_key_iv: "cc5220e835d3a26a0eb4e431b642f127"
server_key_version: "encryptionkey@0"
I20260928 12:21:16.226051 3172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:16.239460 3172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:16.241835 3172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:16.241986 3172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:16.242249 3172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:21:16.242621 3172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:16.242654 3172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:16.242688 3172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:16.242710 3172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:16.248883 3172 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.3:44389
I20260928 12:21:16.248978 3300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.3:44389 every 8 connection(s)
I20260928 12:21:16.249312 3172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
Sep 28 12:21:16 dist-test-slave-920p krb5kdc[2797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598076, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:16.257004 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 3172
I20260928 12:21:16.257107 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance
I20260928 12:21:16.257297 2272 external_mini_cluster.cc:1428] Setting key 41cfbced30f3d07ca630b04082aa59b9
I20260928 12:21:16.259011 2881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:16.250118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:33323 (local address 127.2.56.62:36579)
0928 12:21:16.250185 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:16.250187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:16.250422 (+ 235us) server_negotiation.cc:407] Connection header received
0928 12:21:16.251454 (+ 1032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:16.251455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:16.251467 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:16.251522 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:16.252231 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:16.252771 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:16.253855 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:16.254071 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:16.254123 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:16.256332 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:16.256334 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:16.256335 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:16.256342 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:16.257705 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:16.258115 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:16.258116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:16.258117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:16.258182 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:16.258419 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:16.258420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:16.258421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:16.258559 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:16.258620 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:16.258854 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 12:21:16.258936 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 12:21:16.259203 3303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:16.250103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:36579 (local address 127.2.56.3:33323)
0928 12:21:16.250274 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:16.250323 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:16.251359 (+ 1036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:16.251601 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:16.251605 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:16.251926 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:16.252160 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:16.252165 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:16.252836 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:16.252836 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:16.253788 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:16.253791 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:16.254161 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:16.254162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:16.254284 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:16.254833 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:16.254842 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:16.256242 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:16.257790 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:16.257797 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:16.257799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:16.258050 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:16.258250 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:16.258251 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:16.258251 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:16.258374 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:16.258609 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:16.258615 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:16.258745 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 12:21:16.258836 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 12:21:16.258984 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260928 12:21:16.259665 3301 heartbeater.cc:347] Connected to a master server at 127.2.56.62:36579
I20260928 12:21:16.259766 3301 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:16.259938 3301 heartbeater.cc:511] Master 127.2.56.62:36579 requested a full tablet report, sending...
I20260928 12:21:16.260299 2842 ts_manager.cc:194] Registered new tserver with Master: 9f223427b1ca45a2b9d4557837d0570e (127.2.56.3:44389)
I20260928 12:21:16.260732 2842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.3@KRBTEST.COM'} at 127.2.56.3:33323
I20260928 12:21:16.269246 2272 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:21:16.276470 2881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:16.270999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37888 (local address 127.2.56.62:36579)
0928 12:21:16.271059 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:16.271061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:16.271147 (+ 86us) server_negotiation.cc:407] Connection header received
0928 12:21:16.271214 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:16.271215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:16.271225 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:16.271264 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:16.271810 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:16.272337 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:16.273027 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:16.273207 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:16.273253 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:16.274453 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:16.274455 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:16.274455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:16.274462 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:16.275572 (+ 1110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:16.275854 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:16.275855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:16.275855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:16.275901 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:16.276034 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:16.276035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:16.276035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:16.276184 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:16.276265 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:16.276339 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 12:21:16.276395 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 12:21:16.278921 2842 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:21:16.280944 2842 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:21:16.288256 3316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:16.283690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49894 (local address 127.2.56.2:43627)
0928 12:21:16.284185 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:16.284190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:16.284200 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:21:16.284315 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:16.284317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:16.284333 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:16.284683 (+ 350us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:16.284794 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:16.285676 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:16.286883 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:16.287638 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:16.287878 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:16.287956 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:16.288103 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 12:21:16.288159 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":104,"threads_started":1}
I20260928 12:21:16.288260 3317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:16.283939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39270 (local address 127.2.56.1:35969)
0928 12:21:16.284162 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:16.284168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:16.284180 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:21:16.284228 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:16.284230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:16.284247 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:16.284341 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:16.284590 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:16.285757 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:16.287117 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:16.287840 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:16.287894 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:16.287967 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:16.288142 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 12:21:16.288184 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":98,"threads_started":1}
I20260928 12:21:16.288553 3314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:16.283579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53464 (local address 127.2.56.3:44389)
0928 12:21:16.283817 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:16.283822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:16.284953 (+ 1131us) server_negotiation.cc:407] Connection header received
0928 12:21:16.285038 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:16.285040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:16.285056 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:16.285132 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:16.285228 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:16.286312 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:16.287460 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:16.288073 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:16.288129 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:16.288191 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:16.288409 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 12:21:16.288462 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":94,"threads_started":1}
I20260928 12:21:16.288786 3312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:16.283490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:44389 (local address 127.0.0.1:53464)
0928 12:21:16.284874 (+ 1384us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:16.284885 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:16.284947 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:16.285136 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:16.285137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:16.285139 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:16.285180 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:16.285181 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:16.286418 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:16.286419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:16.287383 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:16.287385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:16.288224 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:16.288225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:16.288282 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:16.288286 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:21:16.288639 (+ 353us) client_negotiation.cc:240] Negotiation successful
0928 12:21:16.288691 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":77,"threads_started":1}
I20260928 12:21:16.289680 3313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:16.283475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:43627 (local address 127.0.0.1:49894)
0928 12:21:16.283630 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:16.283678 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:16.283757 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:16.284442 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:16.284446 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:16.284450 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:16.284550 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:16.284552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:16.285762 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:16.285763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:16.286810 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:16.286813 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:16.287885 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:16.287886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:16.287980 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:16.287998 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 12:21:16.288079 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 12:21:16.289605 (+ 1526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":44,"thread_start_us":71,"threads_started":1}
I20260928 12:21:16.289711 3235 tablet_service.cc:1467] Processing CreateTablet for tablet 84a78b9de5aa4a0ba7a4b1f5199b402f (DEFAULT_TABLE table=test-table [id=8ee1019588d041079362d60fc1146965]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:16.289935 3315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:16.283698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:35969 (local address 127.0.0.1:39270)
0928 12:21:16.283985 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:16.283995 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:16.284051 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:16.284443 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:16.284450 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:16.284452 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:16.284530 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:16.284532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:16.285882 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:16.285884 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:16.287025 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:16.287027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:16.287998 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:16.288000 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:16.288048 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:16.288052 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:21:16.288097 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 12:21:16.289874 (+ 1777us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":1720,"thread_start_us":73,"threads_started":1}
I20260928 12:21:16.290154 3235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84a78b9de5aa4a0ba7a4b1f5199b402f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:16.290616 2967 tablet_service.cc:1467] Processing CreateTablet for tablet 84a78b9de5aa4a0ba7a4b1f5199b402f (DEFAULT_TABLE table=test-table [id=8ee1019588d041079362d60fc1146965]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:16.290942 2967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84a78b9de5aa4a0ba7a4b1f5199b402f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:16.293322 3318 tablet_bootstrap.cc:492] T 84a78b9de5aa4a0ba7a4b1f5199b402f P 9f223427b1ca45a2b9d4557837d0570e: Bootstrap starting.
I20260928 12:21:16.293885 3319 tablet_bootstrap.cc:492] T 84a78b9de5aa4a0ba7a4b1f5199b402f P c51124ccc90547b5a4bfb9a33ed892d4: Bootstrap starting.
I20260928 12:21:16.293840 3101 tablet_service.cc:1467] Processing CreateTablet for tablet 84a78b9de5aa4a0ba7a4b1f5199b402f (DEFAULT_TABLE table=test-table [id=8ee1019588d041079362d60fc1146965]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:16.294109 3318 tablet_bootstrap.cc:654] T 84a78b9de5aa4a0ba7a4b1f5199b402f P 9f223427b1ca45a2b9d4557837d0570e: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:16.294224 3101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84a78b9de5aa4a0ba7a4b1f5199b402f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:16.294688 3318 log.cc:824] T 84a78b9de5aa4a0ba7a4b1f5199b402f P 9f223427b1ca45a2b9d4557837d0570e: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:16.294713 3319 tablet_bootstrap.cc:654] T 84a78b9de5aa4a0ba7a4b1f5199b402f P c51124ccc90547b5a4bfb9a33ed892d4: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:16.295346 3319 log.cc:824] T 84a78b9de5aa4a0ba7a4b1f5199b402f P c51124ccc90547b5a4bfb9a33ed892d4: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:16.295433 3318 tablet_bootstrap.cc:492] T 84a78b9de5aa4a0ba7a4b1f5199b402f P 9f223427b1ca45a2b9d4557837d0570e: No bootstrap required, opened a new log
I20260928 12:21:16.295499 3318 ts_tablet_manager.cc:1397] T 84a78b9de5aa4a0ba7a4b1f5199b402f P 9f223427b1ca45a2b9d4557837d0570e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:21:16.296140 3319 tablet_bootstrap.cc:492] T 84a78b9de5aa4a0ba7a4b1f5199b402f P c51124ccc90547b5a4bfb9a33ed892d4: No bootstrap required, opened a new log
I20260928 12:21:16.296211 3319 ts_tablet_manager.cc:1397] T 84a78b9de5aa4a0ba7a4b1f5199b402f P c51124ccc90547b5a4bfb9a33ed892d4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:21:16.297122 3318 raft_consensus.cc:348] T 84a78b9de5aa4a0ba7a4b1f5199b402f P 9f223427b1ca45a2b9d4557837d0570e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f223427b1ca45a2b9d4557837d0570e" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44389 } } peers { permanent_uuid: "c51124ccc90547b5a4bfb9a33ed892d4" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 35969 } } peers { permanent_uuid: "a358fff20a4d4d31935db08d80fd3409" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 43627 } }
I20260928 12:21:16.297310 3318 raft_consensus.cc:374] T 84a78b9de5aa4a0ba7a4b1f5199b402f P 9f223427b1ca45a2b9d4557837d0570e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:16.297348 3318 raft_consensus.cc:729] T 84a78b9de5aa4a0ba7a4b1f5199b402f P 9f223427b1ca45a2b9d4557837d0570e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f223427b1ca45a2b9d4557837d0570e, State: Initialized, Role: FOLLOWER
I20260928 12:21:16.297490 3318 consensus_queue.cc:260] T 84a78b9de5aa4a0ba7a4b1f5199b402f P 9f223427b1ca45a2b9d4557837d0570e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f223427b1ca45a2b9d4557837d0570e" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44389 } } peers { permanent_uuid: "c51124ccc90547b5a4bfb9a33ed892d4" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 35969 } } peers { permanent_uuid: "a358fff20a4d4d31935db08d80fd3409" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 43627 } }
I20260928 12:21:16.297806 3318 ts_tablet_manager.cc:1428] T 84a78b9de5aa4a0ba7a4b1f5199b402f P 9f223427b1ca45a2b9d4557837d0570e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 12:21:16.297792 3322 tablet_bootstrap.cc:492] T 84a78b9de5aa4a0ba7a4b1f5199b402f P a358fff20a4d4d31935db08d80fd3409: Bootstrap starting.
I20260928 12:21:16.297941 3301 heartbeater.cc:503] Master 127.2.56.62:36579 was elected leader, sending a full tablet report...
I20260928 12:21:16.298517 3322 tablet_bootstrap.cc:654] T 84a78b9de5aa4a0ba7a4b1f5199b402f P a358fff20a4d4d31935db08d80fd3409: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:16.298558 3319 raft_consensus.cc:348] T 84a78b9de5aa4a0ba7a4b1f5199b402f P c51124ccc90547b5a4bfb9a33ed892d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f223427b1ca45a2b9d4557837d0570e" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44389 } } peers { permanent_uuid: "c51124ccc90547b5a4bfb9a33ed892d4" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 35969 } } peers { permanent_uuid: "a358fff20a4d4d31935db08d80fd3409" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 43627 } }
I20260928 12:21:16.298722 3319 raft_consensus.cc:374] T 84a78b9de5aa4a0ba7a4b1f5199b402f P c51124ccc90547b5a4bfb9a33ed892d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:16.298761 3319 raft_consensus.cc:729] T 84a78b9de5aa4a0ba7a4b1f5199b402f P c51124ccc90547b5a4bfb9a33ed892d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c51124ccc90547b5a4bfb9a33ed892d4, State: Initialized, Role: FOLLOWER
I20260928 12:21:16.298877 3319 consensus_queue.cc:260] T 84a78b9de5aa4a0ba7a4b1f5199b402f P c51124ccc90547b5a4bfb9a33ed892d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f223427b1ca45a2b9d4557837d0570e" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44389 } } peers { permanent_uuid: "c51124ccc90547b5a4bfb9a33ed892d4" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 35969 } } peers { permanent_uuid: "a358fff20a4d4d31935db08d80fd3409" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 43627 } }
I20260928 12:21:16.299080 3322 log.cc:824] T 84a78b9de5aa4a0ba7a4b1f5199b402f P a358fff20a4d4d31935db08d80fd3409: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:16.299178 3033 heartbeater.cc:503] Master 127.2.56.62:36579 was elected leader, sending a full tablet report...
I20260928 12:21:16.299177 3319 ts_tablet_manager.cc:1428] T 84a78b9de5aa4a0ba7a4b1f5199b402f P c51124ccc90547b5a4bfb9a33ed892d4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:21:16.300004 3322 tablet_bootstrap.cc:492] T 84a78b9de5aa4a0ba7a4b1f5199b402f P a358fff20a4d4d31935db08d80fd3409: No bootstrap required, opened a new log
I20260928 12:21:16.300108 3322 ts_tablet_manager.cc:1397] T 84a78b9de5aa4a0ba7a4b1f5199b402f P a358fff20a4d4d31935db08d80fd3409: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:21:16.301592 3322 raft_consensus.cc:348] T 84a78b9de5aa4a0ba7a4b1f5199b402f P a358fff20a4d4d31935db08d80fd3409 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f223427b1ca45a2b9d4557837d0570e" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44389 } } peers { permanent_uuid: "c51124ccc90547b5a4bfb9a33ed892d4" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 35969 } } peers { permanent_uuid: "a358fff20a4d4d31935db08d80fd3409" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 43627 } }
I20260928 12:21:16.301729 3322 raft_consensus.cc:374] T 84a78b9de5aa4a0ba7a4b1f5199b402f P a358fff20a4d4d31935db08d80fd3409 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:16.301750 3322 raft_consensus.cc:729] T 84a78b9de5aa4a0ba7a4b1f5199b402f P a358fff20a4d4d31935db08d80fd3409 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a358fff20a4d4d31935db08d80fd3409, State: Initialized, Role: FOLLOWER
I20260928 12:21:16.301837 3322 consensus_queue.cc:260] T 84a78b9de5aa4a0ba7a4b1f5199b402f P a358fff20a4d4d31935db08d80fd3409 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f223427b1ca45a2b9d4557837d0570e" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44389 } } peers { permanent_uuid: "c51124ccc90547b5a4bfb9a33ed892d4" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 35969 } } peers { permanent_uuid: "a358fff20a4d4d31935db08d80fd3409" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 43627 } }
I20260928 12:21:16.302049 3322 ts_tablet_manager.cc:1428] T 84a78b9de5aa4a0ba7a4b1f5199b402f P a358fff20a4d4d31935db08d80fd3409: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:21:16.302081 3167 heartbeater.cc:503] Master 127.2.56.62:36579 was elected leader, sending a full tablet report...
I20260928 12:21:16.324438 3324 raft_consensus.cc:482] T 84a78b9de5aa4a0ba7a4b1f5199b402f P c51124ccc90547b5a4bfb9a33ed892d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:21:16.324610 3324 raft_consensus.cc:504] T 84a78b9de5aa4a0ba7a4b1f5199b402f P c51124ccc90547b5a4bfb9a33ed892d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f223427b1ca45a2b9d4557837d0570e" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44389 } } peers { permanent_uuid: "c51124ccc90547b5a4bfb9a33ed892d4" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 35969 } } peers { permanent_uuid: "a358fff20a4d4d31935db08d80fd3409" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 43627 } }
I20260928 12:21:16.325101 3324 leader_election.cc:288] T 84a78b9de5aa4a0ba7a4b1f5199b402f P c51124ccc90547b5a4bfb9a33ed892d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f223427b1ca45a2b9d4557837d0570e (127.2.56.3:44389), a358fff20a4d4d31935db08d80fd3409 (127.2.56.2:43627)
W20260928 12:21:16.328513 3168 tablet.cc:2369] T 84a78b9de5aa4a0ba7a4b1f5199b402f P a358fff20a4d4d31935db08d80fd3409: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:21:16.329062 3035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:16.325299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:43627 (local address 127.2.56.1:33703)
0928 12:21:16.325347 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:16.325360 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:16.325407 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:16.325625 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:16.325627 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:16.325630 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:16.325722 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:16.325725 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:16.326673 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:16.326674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:16.327990 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:16.327993 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:16.328773 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:16.328774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:16.328850 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:16.328854 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:21:16.328898 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 12:21:16.328951 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":28}
I20260928 12:21:16.329073 3316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:16.325392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:33703 (local address 127.2.56.2:43627)
0928 12:21:16.325441 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:16.325443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:16.325454 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:21:16.325506 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:16.325507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:16.325521 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:16.325585 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:16.325798 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:16.326597 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:16.328086 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:16.328654 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:16.328726 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:16.328739 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:16.328917 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 12:21:16.328963 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:21:16.329090 3314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:16.325375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:41891 (local address 127.2.56.3:44389)
0928 12:21:16.325428 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:16.325431 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:16.325709 (+ 278us) server_negotiation.cc:407] Connection header received
0928 12:21:16.325769 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:16.325770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:16.325786 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:16.325841 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:16.325961 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:16.326666 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:16.328080 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:16.328683 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:16.328734 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:16.328743 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:16.328956 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 12:21:16.329001 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 12:21:16.329289 3327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:16.325299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:44389 (local address 127.2.56.1:41891)
0928 12:21:16.325619 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:16.325629 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:16.325707 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:16.325848 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:16.325849 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:16.325850 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:16.325907 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:16.325908 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:16.326730 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:16.326731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:16.327990 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:16.327993 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:16.328764 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:16.328765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:16.328879 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:16.328882 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:21:16.328934 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 12:21:16.328975 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":75,"thread_start_us":148,"threads_started":1}
I20260928 12:21:16.329483 3121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84a78b9de5aa4a0ba7a4b1f5199b402f" candidate_uuid: "c51124ccc90547b5a4bfb9a33ed892d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a358fff20a4d4d31935db08d80fd3409" is_pre_election: true
I20260928 12:21:16.329675 3121 raft_consensus.cc:2454] T 84a78b9de5aa4a0ba7a4b1f5199b402f P a358fff20a4d4d31935db08d80fd3409 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c51124ccc90547b5a4bfb9a33ed892d4 in term 0.
I20260928 12:21:16.329928 2921 leader_election.cc:302] T 84a78b9de5aa4a0ba7a4b1f5199b402f P c51124ccc90547b5a4bfb9a33ed892d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a358fff20a4d4d31935db08d80fd3409, c51124ccc90547b5a4bfb9a33ed892d4; no voters:
I20260928 12:21:16.330016 3255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84a78b9de5aa4a0ba7a4b1f5199b402f" candidate_uuid: "c51124ccc90547b5a4bfb9a33ed892d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f223427b1ca45a2b9d4557837d0570e" is_pre_election: true
I20260928 12:21:16.330088 3324 raft_consensus.cc:2781] T 84a78b9de5aa4a0ba7a4b1f5199b402f P c51124ccc90547b5a4bfb9a33ed892d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:21:16.330138 3324 raft_consensus.cc:482] T 84a78b9de5aa4a0ba7a4b1f5199b402f P c51124ccc90547b5a4bfb9a33ed892d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:21:16.330163 3324 raft_consensus.cc:3037] T 84a78b9de5aa4a0ba7a4b1f5199b402f P c51124ccc90547b5a4bfb9a33ed892d4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:16.330204 3255 raft_consensus.cc:2454] T 84a78b9de5aa4a0ba7a4b1f5199b402f P 9f223427b1ca45a2b9d4557837d0570e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c51124ccc90547b5a4bfb9a33ed892d4 in term 0.
I20260928 12:21:16.331024 3324 raft_consensus.cc:504] T 84a78b9de5aa4a0ba7a4b1f5199b402f P c51124ccc90547b5a4bfb9a33ed892d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f223427b1ca45a2b9d4557837d0570e" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44389 } } peers { permanent_uuid: "c51124ccc90547b5a4bfb9a33ed892d4" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 35969 } } peers { permanent_uuid: "a358fff20a4d4d31935db08d80fd3409" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 43627 } }
I20260928 12:21:16.331158 3324 leader_election.cc:288] T 84a78b9de5aa4a0ba7a4b1f5199b402f P c51124ccc90547b5a4bfb9a33ed892d4 [CANDIDATE]: Term 1 election: Requested vote from peers 9f223427b1ca45a2b9d4557837d0570e (127.2.56.3:44389), a358fff20a4d4d31935db08d80fd3409 (127.2.56.2:43627)
I20260928 12:21:16.331291 3121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84a78b9de5aa4a0ba7a4b1f5199b402f" candidate_uuid: "c51124ccc90547b5a4bfb9a33ed892d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a358fff20a4d4d31935db08d80fd3409"
I20260928 12:21:16.331313 3255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84a78b9de5aa4a0ba7a4b1f5199b402f" candidate_uuid: "c51124ccc90547b5a4bfb9a33ed892d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f223427b1ca45a2b9d4557837d0570e"
I20260928 12:21:16.331382 3255 raft_consensus.cc:3037] T 84a78b9de5aa4a0ba7a4b1f5199b402f P 9f223427b1ca45a2b9d4557837d0570e [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:16.331377 3121 raft_consensus.cc:3037] T 84a78b9de5aa4a0ba7a4b1f5199b402f P a358fff20a4d4d31935db08d80fd3409 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:16.332177 3121 raft_consensus.cc:2454] T 84a78b9de5aa4a0ba7a4b1f5199b402f P a358fff20a4d4d31935db08d80fd3409 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c51124ccc90547b5a4bfb9a33ed892d4 in term 1.
I20260928 12:21:16.332181 3255 raft_consensus.cc:2454] T 84a78b9de5aa4a0ba7a4b1f5199b402f P 9f223427b1ca45a2b9d4557837d0570e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c51124ccc90547b5a4bfb9a33ed892d4 in term 1.
I20260928 12:21:16.332334 2921 leader_election.cc:302] T 84a78b9de5aa4a0ba7a4b1f5199b402f P c51124ccc90547b5a4bfb9a33ed892d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a358fff20a4d4d31935db08d80fd3409, c51124ccc90547b5a4bfb9a33ed892d4; no voters:
I20260928 12:21:16.332451 3324 raft_consensus.cc:2781] T 84a78b9de5aa4a0ba7a4b1f5199b402f P c51124ccc90547b5a4bfb9a33ed892d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:21:16.332602 3324 raft_consensus.cc:686] T 84a78b9de5aa4a0ba7a4b1f5199b402f P c51124ccc90547b5a4bfb9a33ed892d4 [term 1 LEADER]: Becoming Leader. State: Replica: c51124ccc90547b5a4bfb9a33ed892d4, State: Running, Role: LEADER
I20260928 12:21:16.332696 3324 consensus_queue.cc:237] T 84a78b9de5aa4a0ba7a4b1f5199b402f P c51124ccc90547b5a4bfb9a33ed892d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f223427b1ca45a2b9d4557837d0570e" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44389 } } peers { permanent_uuid: "c51124ccc90547b5a4bfb9a33ed892d4" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 35969 } } peers { permanent_uuid: "a358fff20a4d4d31935db08d80fd3409" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 43627 } }
I20260928 12:21:16.333637 2841 catalog_manager.cc:5495] T 84a78b9de5aa4a0ba7a4b1f5199b402f P c51124ccc90547b5a4bfb9a33ed892d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to c51124ccc90547b5a4bfb9a33ed892d4 (127.2.56.1). New cstate: current_term: 1 leader_uuid: "c51124ccc90547b5a4bfb9a33ed892d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f223427b1ca45a2b9d4557837d0570e" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c51124ccc90547b5a4bfb9a33ed892d4" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 35969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a358fff20a4d4d31935db08d80fd3409" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 43627 } health_report { overall_health: UNKNOWN } } }
Sep 28 12:21:16 dist-test-slave-920p krb5kdc[2797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.1@KRBTEST.COM
I20260928 12:21:16.350272 3317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:16.342711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39284 (local address 127.2.56.1:35969)
0928 12:21:16.342770 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:16.342772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:16.342882 (+ 110us) server_negotiation.cc:407] Connection header received
0928 12:21:16.342949 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:16.342950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:16.342964 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:16.343016 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:16.343612 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:16.344129 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:16.344846 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:16.345051 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:16.345107 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:16.347484 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:16.347499 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:16.347499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:16.347507 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:16.349104 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:16.349481 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:16.349485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:16.349486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:16.349555 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:16.349715 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:16.349716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:16.349716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:16.349882 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:16.350042 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:16.350076 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 12:21:16.350162 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
Sep 28 12:21:16 dist-test-slave-920p krb5kdc[2797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598076, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 12:21:16 dist-test-slave-920p krb5kdc[2797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598076, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.56.1@KRBTEST.COM
I20260928 12:21:16.373176 3317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:16.366236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39296 (local address 127.2.56.1:35969)
0928 12:21:16.366303 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:16.366305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:16.366393 (+ 88us) server_negotiation.cc:407] Connection header received
0928 12:21:16.366483 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:16.366484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:16.366496 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:16.366576 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:16.367138 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:16.367682 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:16.368420 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:16.368629 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:16.368686 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:16.370995 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:16.370996 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:16.370997 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:16.371005 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:16.372273 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:16.372547 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:16.372548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:16.372548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:16.372594 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:16.372734 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:16.372735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:16.372735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:16.372822 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:16.372909 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:16.373034 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 12:21:16.373098 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Sep 28 12:21:16 dist-test-slave-920p krb5kdc[2797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 12:21:16 dist-test-slave-920p krb5kdc[2797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.1@KRBTEST.COM
I20260928 12:21:16.395605 3317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:16.388652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39312 (local address 127.2.56.1:35969)
0928 12:21:16.388718 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:16.388720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:16.388793 (+ 73us) server_negotiation.cc:407] Connection header received
0928 12:21:16.388880 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:16.388881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:16.388892 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:16.388938 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:16.389533 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:16.390048 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:16.390787 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:16.390987 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:16.391043 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:16.393250 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:16.393252 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:16.393253 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:16.393301 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:16.394600 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:16.394915 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:16.394917 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:16.394917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:16.394971 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:16.395127 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:16.395128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:16.395129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:16.395231 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:16.395320 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:16.395421 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 12:21:16.395500 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260928 12:21:16.396123 2947 tablet_service.cc:2908] Rejecting scan request for tablet 84a78b9de5aa4a0ba7a4b1f5199b402f: Uninitialized: clean time has not yet been initialized
I20260928 12:21:16.397001 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 2904
I20260928 12:21:16.402526 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 3038
I20260928 12:21:16.406946 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 3172
I20260928 12:21:16.411074 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 2810
2026-09-28T12:21:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1891 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:21:16 dist-test-slave-920p krb5kdc[3342](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:21:16 dist-test-slave-920p krb5kdc[3342](info): set up 2 sockets
Sep 28 12:21:16 dist-test-slave-920p krb5kdc[3342](info): commencing operation
krb5kdc: starting...
W20260928 12:21:17.373200 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:21:17 dist-test-slave-920p krb5kdc[3342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:21:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:21:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
I20260928 12:21:17.490299 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:37841
--webserver_interface=127.2.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:33593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:37841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:17.578373 3355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:17.578616 3355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:17.578639 3355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:17.579873 3355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:21:17.579924 3355 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 12:21:17.579936 3355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:17.579947 3355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:17.579958 3355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:21:17.579974 3355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:21:17.581614 3355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:33593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:37841
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:37841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:17.581889 3355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:17.582137 3355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:17.587327 3361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:17.587500 3355 server_base.cc:1034] running on GCE node
W20260928 12:21:17.587360 3363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:17.587327 3360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:17.587977 3355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:17.588313 3355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:17.589466 3355 hybrid_clock.cc:648] HybridClock initialized: now 1790598077589442 us; error 33 us; skew 500 ppm
Sep 28 12:21:17 dist-test-slave-920p krb5kdc[3342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598077, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:17.592342 3355 init.cc:377] Logged in from keytab as kudu/127.2.56.62@KRBTEST.COM (short username kudu)
I20260928 12:21:17.592847 3355 webserver.cc:460] Webserver started at http://127.2.56.62:36687/ using document root <none> and password file <none>
I20260928 12:21:17.593092 3355 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:17.593142 3355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:17.593303 3355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:17.594588 3355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "628ade5983aa468a9f6246ca00163a30"
format_stamp: "Formatted at 2026-09-28 12:21:17 on dist-test-slave-920p"
server_key: "fbc600409228440d3b5b5dc4bc2a6a48"
server_key_iv: "ab84c00385b5102f3b585b87d3f1aaa6"
server_key_version: "encryptionkey@0"
I20260928 12:21:17.594957 3355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "628ade5983aa468a9f6246ca00163a30"
format_stamp: "Formatted at 2026-09-28 12:21:17 on dist-test-slave-920p"
server_key: "fbc600409228440d3b5b5dc4bc2a6a48"
server_key_iv: "ab84c00385b5102f3b585b87d3f1aaa6"
server_key_version: "encryptionkey@0"
I20260928 12:21:17.596344 3355 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:21:17.597157 3370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:17.597359 3355 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:21:17.597443 3355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "628ade5983aa468a9f6246ca00163a30"
format_stamp: "Formatted at 2026-09-28 12:21:17 on dist-test-slave-920p"
server_key: "fbc600409228440d3b5b5dc4bc2a6a48"
server_key_iv: "ab84c00385b5102f3b585b87d3f1aaa6"
server_key_version: "encryptionkey@0"
I20260928 12:21:17.597527 3355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:17.615042 3355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:17.618042 3355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:17.618211 3355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:17.624081 3355 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:37841
I20260928 12:21:17.624145 3432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:37841 every 8 connection(s)
I20260928 12:21:17.624578 3355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:21:17.625231 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 3355
I20260928 12:21:17.625361 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:21:17.625553 2272 external_mini_cluster.cc:1428] Setting key d1ec2a6ab8026e27117177ee96004062
I20260928 12:21:17.625571 3433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:17.629760 3433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 628ade5983aa468a9f6246ca00163a30: Bootstrap starting.
I20260928 12:21:17.630669 3433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 628ade5983aa468a9f6246ca00163a30: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:17.630985 3433 log.cc:824] T 00000000000000000000000000000000 P 628ade5983aa468a9f6246ca00163a30: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:17.631922 3433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 628ade5983aa468a9f6246ca00163a30: No bootstrap required, opened a new log
Sep 28 12:21:17 dist-test-slave-920p krb5kdc[3342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:17.634583 3433 raft_consensus.cc:348] T 00000000000000000000000000000000 P 628ade5983aa468a9f6246ca00163a30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628ade5983aa468a9f6246ca00163a30" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 37841 } }
I20260928 12:21:17.634848 3433 raft_consensus.cc:374] T 00000000000000000000000000000000 P 628ade5983aa468a9f6246ca00163a30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:17.634899 3433 raft_consensus.cc:729] T 00000000000000000000000000000000 P 628ade5983aa468a9f6246ca00163a30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 628ade5983aa468a9f6246ca00163a30, State: Initialized, Role: FOLLOWER
I20260928 12:21:17.635092 3433 consensus_queue.cc:260] T 00000000000000000000000000000000 P 628ade5983aa468a9f6246ca00163a30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628ade5983aa468a9f6246ca00163a30" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 37841 } }
I20260928 12:21:17.635174 3433 raft_consensus.cc:388] T 00000000000000000000000000000000 P 628ade5983aa468a9f6246ca00163a30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:21:17.635205 3433 raft_consensus.cc:482] T 00000000000000000000000000000000 P 628ade5983aa468a9f6246ca00163a30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:21:17.635259 3433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 628ade5983aa468a9f6246ca00163a30 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:17.636009 3433 raft_consensus.cc:504] T 00000000000000000000000000000000 P 628ade5983aa468a9f6246ca00163a30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628ade5983aa468a9f6246ca00163a30" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 37841 } }
I20260928 12:21:17.636122 3433 leader_election.cc:302] T 00000000000000000000000000000000 P 628ade5983aa468a9f6246ca00163a30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 628ade5983aa468a9f6246ca00163a30; no voters:
I20260928 12:21:17.636332 3433 leader_election.cc:288] T 00000000000000000000000000000000 P 628ade5983aa468a9f6246ca00163a30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:21:17.636423 3438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 628ade5983aa468a9f6246ca00163a30 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:21:17.636585 3438 raft_consensus.cc:686] T 00000000000000000000000000000000 P 628ade5983aa468a9f6246ca00163a30 [term 1 LEADER]: Becoming Leader. State: Replica: 628ade5983aa468a9f6246ca00163a30, State: Running, Role: LEADER
I20260928 12:21:17.636710 3433 sys_catalog.cc:564] T 00000000000000000000000000000000 P 628ade5983aa468a9f6246ca00163a30 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:21:17.636688 3438 consensus_queue.cc:237] T 00000000000000000000000000000000 P 628ade5983aa468a9f6246ca00163a30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628ade5983aa468a9f6246ca00163a30" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 37841 } }
I20260928 12:21:17.637112 3438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 628ade5983aa468a9f6246ca00163a30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 628ade5983aa468a9f6246ca00163a30. Latest consensus state: current_term: 1 leader_uuid: "628ade5983aa468a9f6246ca00163a30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628ade5983aa468a9f6246ca00163a30" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 37841 } } }
I20260928 12:21:17.637192 3438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 628ade5983aa468a9f6246ca00163a30 [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:17.637454 3439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 628ade5983aa468a9f6246ca00163a30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "628ade5983aa468a9f6246ca00163a30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628ade5983aa468a9f6246ca00163a30" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 37841 } } }
I20260928 12:21:17.637504 3442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:21:17.637526 3439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 628ade5983aa468a9f6246ca00163a30 [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:17.638388 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:17.626177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49402 (local address 127.2.56.62:37841)
0928 12:21:17.626387 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:17.626398 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:17.626582 (+ 184us) server_negotiation.cc:407] Connection header received
0928 12:21:17.627092 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:17.627102 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:17.627169 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:17.627331 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:17.628230 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:17.629333 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:17.630430 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:17.630965 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:17.631064 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:17.634210 (+ 3146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:17.634217 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:17.634217 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:17.634234 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:17.636316 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:17.636855 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:17.636860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:17.636861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:17.636985 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:17.637194 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:17.637196 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:17.637197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:17.637635 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:17.637826 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:17.637953 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 12:21:17.638117 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":104,"threads_started":1}
I20260928 12:21:17.638422 3442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:21:17.640529 3442 catalog_manager.cc:1347] Generated new cluster ID: 791c309e48834b78b8c2c616c4bb9cc9
I20260928 12:21:17.640592 3442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:21:17.649818 3442 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:21:17.650537 3442 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:21:17.658272 3442 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 628ade5983aa468a9f6246ca00163a30: Generated new TSK 0
I20260928 12:21:17.658614 3442 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab.
Entry for principal kudu/127.2.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab.
I20260928 12:21:17.687815 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.1:0
--local_ip_for_outbound_sockets=127.2.56.1
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:37841
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:33593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:17.778224 3459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:17.778475 3459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:17.778499 3459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:17.779736 3459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:17.779780 3459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:17.779800 3459 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 12:21:17.779830 3459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.1
I20260928 12:21:17.781358 3459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:33593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.56.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.56.62:37841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.1
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:17.781636 3459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:17.781883 3459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:17.787467 3467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:17.787518 3465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:17.787461 3464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:17.787695 3459 server_base.cc:1034] running on GCE node
I20260928 12:21:17.787986 3459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:17.788240 3459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:17.789386 3459 hybrid_clock.cc:648] HybridClock initialized: now 1790598077789363 us; error 35 us; skew 500 ppm
Sep 28 12:21:17 dist-test-slave-920p krb5kdc[3342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598077, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:17.792052 3459 init.cc:377] Logged in from keytab as kudu/127.2.56.1@KRBTEST.COM (short username kudu)
I20260928 12:21:17.792543 3459 webserver.cc:460] Webserver started at http://127.2.56.1:39371/ using document root <none> and password file <none>
I20260928 12:21:17.792789 3459 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:17.792841 3459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:17.792976 3459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:17.794240 3459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-0/data/instance:
uuid: "9248462f96c0451a9b54e19225f3dd0e"
format_stamp: "Formatted at 2026-09-28 12:21:17 on dist-test-slave-920p"
server_key: "3b8324468850a611716ef1ba61a07be4"
server_key_iv: "bc25ee17403e90c1a65482fe08641f84"
server_key_version: "encryptionkey@0"
I20260928 12:21:17.794597 3459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance:
uuid: "9248462f96c0451a9b54e19225f3dd0e"
format_stamp: "Formatted at 2026-09-28 12:21:17 on dist-test-slave-920p"
server_key: "3b8324468850a611716ef1ba61a07be4"
server_key_iv: "bc25ee17403e90c1a65482fe08641f84"
server_key_version: "encryptionkey@0"
I20260928 12:21:17.795922 3459 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 12:21:17.796581 3474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:17.796733 3459 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:21:17.796816 3459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-0/wal
uuid: "9248462f96c0451a9b54e19225f3dd0e"
format_stamp: "Formatted at 2026-09-28 12:21:17 on dist-test-slave-920p"
server_key: "3b8324468850a611716ef1ba61a07be4"
server_key_iv: "bc25ee17403e90c1a65482fe08641f84"
server_key_version: "encryptionkey@0"
I20260928 12:21:17.796891 3459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:17.817126 3459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:17.819471 3459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:17.819630 3459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:17.820250 3459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:17.820297 3459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:17.820329 3459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:17.820346 3459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:17.827569 3459 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.1:36457
I20260928 12:21:17.827627 3589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.1:36457 every 8 connection(s)
Sep 28 12:21:17 dist-test-slave-920p krb5kdc[3342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598077, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:17.828042 3459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
I20260928 12:21:17.830758 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:17.820786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:41265 (local address 127.2.56.62:37841)
0928 12:21:17.820859 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:17.820863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:17.821070 (+ 207us) server_negotiation.cc:407] Connection header received
0928 12:21:17.821716 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:17.821717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:17.821733 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:17.821813 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:17.822683 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:17.823524 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:17.825176 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:17.825549 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:17.825641 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:17.828403 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:17.828405 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:17.828405 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:17.828414 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:17.829610 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:17.829997 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:17.829998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:17.829999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:17.830047 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:17.830223 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:17.830224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:17.830224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:17.830330 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:17.830379 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:17.830605 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 12:21:17.830682 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 12:21:17.830940 3483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:17.820746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:37841 (local address 127.2.56.1:41265)
0928 12:21:17.820927 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:17.820974 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:17.821602 (+ 628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:17.821909 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:17.821914 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:17.822241 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:17.822571 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:17.822578 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:17.823635 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:17.823637 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:17.825072 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:17.825075 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:17.825646 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:17.825647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:17.825799 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:17.826345 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:17.826367 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:17.828287 (+ 1920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:17.829688 (+ 1401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:17.829695 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:17.829697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:17.829939 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:17.830091 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:17.830092 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:17.830092 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:17.830183 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:17.830370 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:17.830373 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:17.830504 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 12:21:17.830590 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 12:21:17.830720 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
I20260928 12:21:17.831446 3590 heartbeater.cc:347] Connected to a master server at 127.2.56.62:37841
I20260928 12:21:17.831529 3590 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:17.831736 3590 heartbeater.cc:511] Master 127.2.56.62:37841 requested a full tablet report, sending...
I20260928 12:21:17.832346 3387 ts_manager.cc:194] Registered new tserver with Master: 9248462f96c0451a9b54e19225f3dd0e (127.2.56.1:36457)
I20260928 12:21:17.833233 3387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.1@KRBTEST.COM'} at 127.2.56.1:41265
I20260928 12:21:17.833617 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 3459
I20260928 12:21:17.833714 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance
I20260928 12:21:17.833915 2272 external_mini_cluster.cc:1428] Setting key 11a90e6ca27a8c3b5b44db904b8a51ce
I20260928 12:21:17.838376 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:17.832645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:45831 (local address 127.2.56.62:37841)
0928 12:21:17.832684 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:17.832686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:17.832779 (+ 93us) server_negotiation.cc:407] Connection header received
0928 12:21:17.832831 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:17.832832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:17.832843 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:17.832893 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:17.833162 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:17.833889 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:17.834897 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:17.835105 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:17.835173 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:17.836043 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:17.836045 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:17.836045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:17.836052 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:17.837338 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:17.837679 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:17.837681 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:17.837681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:17.837740 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:17.837894 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:17.837895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:17.837895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:17.838002 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:17.838059 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:17.838205 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 12:21:17.838280 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":9,"server-negotiator.queue_time_us":18}
I20260928 12:21:17.838392 3596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:17.832573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:37841 (local address 127.2.56.1:45831)
0928 12:21:17.832715 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:17.832725 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:17.832780 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:17.832895 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:17.832896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:17.833056 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:17.833112 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:17.833113 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:17.834011 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:17.834012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:17.834799 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:17.834802 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:17.835175 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:17.835176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:17.835254 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:17.835666 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:17.835671 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:17.835967 (+ 296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:17.837419 (+ 1452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:17.837420 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:17.837421 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:17.837625 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:17.837793 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:17.837794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:17.837794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:17.837848 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:17.838072 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:17.838073 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:17.838120 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 12:21:17.838176 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 12:21:17.838282 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":69,"threads_started":1}
WARNING: no policy specified for kudu/127.2.56.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab.
Entry for principal kudu/127.2.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab.
I20260928 12:21:17.860116 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.2:0
--local_ip_for_outbound_sockets=127.2.56.2
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:37841
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:33593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:17.949069 3599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:17.949325 3599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:17.949352 3599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:17.950582 3599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:17.950628 3599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:17.950649 3599 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 12:21:17.950664 3599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.2
I20260928 12:21:17.952181 3599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:33593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.56.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.56.62:37841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.2
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:17.952417 3599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:17.952661 3599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:17.957958 3607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:17.958019 3604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:17.957964 3605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:17.958267 3599 server_base.cc:1034] running on GCE node
I20260928 12:21:17.958499 3599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:17.958735 3599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:17.959877 3599 hybrid_clock.cc:648] HybridClock initialized: now 1790598077959862 us; error 30 us; skew 500 ppm
Sep 28 12:21:17 dist-test-slave-920p krb5kdc[3342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598077, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:17.962520 3599 init.cc:377] Logged in from keytab as kudu/127.2.56.2@KRBTEST.COM (short username kudu)
I20260928 12:21:17.962966 3599 webserver.cc:460] Webserver started at http://127.2.56.2:46227/ using document root <none> and password file <none>
I20260928 12:21:17.963181 3599 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:17.963229 3599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:17.963363 3599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:17.964556 3599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-1/data/instance:
uuid: "604a530966a44b44a5880c3cae2b2115"
format_stamp: "Formatted at 2026-09-28 12:21:17 on dist-test-slave-920p"
server_key: "c44e77d11185cf6da8c2e44c424eee85"
server_key_iv: "fa24ebb5e3f29e757c96fed80ed7c8a2"
server_key_version: "encryptionkey@0"
I20260928 12:21:17.964908 3599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance:
uuid: "604a530966a44b44a5880c3cae2b2115"
format_stamp: "Formatted at 2026-09-28 12:21:17 on dist-test-slave-920p"
server_key: "c44e77d11185cf6da8c2e44c424eee85"
server_key_iv: "fa24ebb5e3f29e757c96fed80ed7c8a2"
server_key_version: "encryptionkey@0"
I20260928 12:21:17.966260 3599 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:21:17.967008 3614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:17.967152 3599 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:17.967236 3599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-1/wal
uuid: "604a530966a44b44a5880c3cae2b2115"
format_stamp: "Formatted at 2026-09-28 12:21:17 on dist-test-slave-920p"
server_key: "c44e77d11185cf6da8c2e44c424eee85"
server_key_iv: "fa24ebb5e3f29e757c96fed80ed7c8a2"
server_key_version: "encryptionkey@0"
I20260928 12:21:17.967298 3599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:17.980813 3599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:17.982748 3599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:17.982905 3599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:17.983525 3599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:17.983572 3599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:17.983604 3599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:17.983619 3599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 12:21:17 dist-test-slave-920p krb5kdc[3342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598077, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:17.991427 3599 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.2:40655
I20260928 12:21:17.991523 3729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.2:40655 every 8 connection(s)
I20260928 12:21:17.991952 3599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
I20260928 12:21:17.994211 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:17.984008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:40951 (local address 127.2.56.62:37841)
0928 12:21:17.984073 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:17.984075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:17.984304 (+ 229us) server_negotiation.cc:407] Connection header received
0928 12:21:17.984900 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:17.984901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:17.984916 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:17.984993 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:17.985799 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:17.986643 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:17.988164 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:17.988469 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:17.988561 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:17.991288 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:17.991290 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:17.991290 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:17.991298 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:17.992946 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:17.993422 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:17.993422 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:17.993423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:17.993481 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:17.993655 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:17.993656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:17.993656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:17.993765 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:17.993809 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:17.994070 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 12:21:17.994138 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 12:21:17.994380 3623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:17.983991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:37841 (local address 127.2.56.2:40951)
0928 12:21:17.984165 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:17.984210 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:17.984790 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:17.985089 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:17.985094 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:17.985451 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:17.985704 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:17.985711 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:17.986743 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:17.986744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:17.988075 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:17.988079 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:17.988571 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:17.988571 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:17.988719 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:17.989357 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:17.989369 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:17.991173 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:17.993051 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:17.993057 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:17.993059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:17.993346 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:17.993521 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:17.993521 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:17.993522 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:17.993619 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:17.993807 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:17.993810 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:17.993961 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 12:21:17.994042 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 12:21:17.994177 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":88,"threads_started":1}
I20260928 12:21:17.994843 3730 heartbeater.cc:347] Connected to a master server at 127.2.56.62:37841
I20260928 12:21:17.994935 3730 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:17.995128 3730 heartbeater.cc:511] Master 127.2.56.62:37841 requested a full tablet report, sending...
I20260928 12:21:17.995532 3383 ts_manager.cc:194] Registered new tserver with Master: 604a530966a44b44a5880c3cae2b2115 (127.2.56.2:40655)
I20260928 12:21:17.995599 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 3599
I20260928 12:21:17.995690 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance
I20260928 12:21:17.995879 2272 external_mini_cluster.cc:1428] Setting key ee645dfb3bafe54782e8ce666864c4af
I20260928 12:21:17.996073 3383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.2@KRBTEST.COM'} at 127.2.56.2:40951
I20260928 12:21:18.002246 3736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:17.995999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:37841 (local address 127.2.56.2:60485)
0928 12:21:17.996325 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:17.996336 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:17.996413 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:17.996528 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:17.996529 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:17.996745 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:17.996826 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:17.996828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:17.997589 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:17.997590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:17.998107 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:17.998108 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:17.998483 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:17.998484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:17.998538 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:17.998988 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:17.998994 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:17.999396 (+ 402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:18.001122 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:18.001124 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:18.001125 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:18.001389 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:18.001576 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:18.001577 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:18.001577 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:18.001640 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:18.001907 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:18.001908 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:18.001955 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 12:21:18.002021 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 12:21:18.002131 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":258,"threads_started":1}
I20260928 12:21:18.002236 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:17.995986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:60485 (local address 127.2.56.62:37841)
0928 12:21:17.996029 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:17.996031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:17.996387 (+ 356us) server_negotiation.cc:407] Connection header received
0928 12:21:17.996464 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:17.996465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:17.996473 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:17.996515 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:17.996885 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:17.997500 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:17.998171 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:17.998421 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:17.998478 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:17.999469 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:17.999470 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:17.999470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:17.999476 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:18.001028 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:18.001462 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:18.001464 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:18.001464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:18.001523 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:18.001688 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:18.001689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:18.001690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:18.001848 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:18.001920 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:18.002050 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 12:21:18.002130 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
WARNING: no policy specified for kudu/127.2.56.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab.
Entry for principal kudu/127.2.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab.
I20260928 12:21:18.023516 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.3:0
--local_ip_for_outbound_sockets=127.2.56.3
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:37841
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:33593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:18.117257 3739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:18.117565 3739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:18.117599 3739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:18.118827 3739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:18.118871 3739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:18.118908 3739 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 12:21:18.118933 3739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.3
I20260928 12:21:18.120464 3739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:33593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.56.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.56.62:37841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.3
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:18.120733 3739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:18.120975 3739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:18.126067 3745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:18.126171 3744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:18.126261 3747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:18.126379 3739 server_base.cc:1034] running on GCE node
I20260928 12:21:18.126752 3739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:18.127058 3739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:18.128214 3739 hybrid_clock.cc:648] HybridClock initialized: now 1790598078128201 us; error 33 us; skew 500 ppm
Sep 28 12:21:18 dist-test-slave-920p krb5kdc[3342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598078, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:18.130977 3739 init.cc:377] Logged in from keytab as kudu/127.2.56.3@KRBTEST.COM (short username kudu)
I20260928 12:21:18.131497 3739 webserver.cc:460] Webserver started at http://127.2.56.3:33689/ using document root <none> and password file <none>
I20260928 12:21:18.131772 3739 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:18.131821 3739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:18.131944 3739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:18.133153 3739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-2/data/instance:
uuid: "72e5033472634cb88da6f701c6aeb7b3"
format_stamp: "Formatted at 2026-09-28 12:21:18 on dist-test-slave-920p"
server_key: "0a4b336096d4fbd4136bccccb8913d82"
server_key_iv: "4a279ae0c0fe74b8ac4d11b38afcaae7"
server_key_version: "encryptionkey@0"
I20260928 12:21:18.133569 3739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance:
uuid: "72e5033472634cb88da6f701c6aeb7b3"
format_stamp: "Formatted at 2026-09-28 12:21:18 on dist-test-slave-920p"
server_key: "0a4b336096d4fbd4136bccccb8913d82"
server_key_iv: "4a279ae0c0fe74b8ac4d11b38afcaae7"
server_key_version: "encryptionkey@0"
I20260928 12:21:18.134887 3739 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:21:18.135650 3754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:18.135797 3739 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:18.135911 3739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-2/wal
uuid: "72e5033472634cb88da6f701c6aeb7b3"
format_stamp: "Formatted at 2026-09-28 12:21:18 on dist-test-slave-920p"
server_key: "0a4b336096d4fbd4136bccccb8913d82"
server_key_iv: "4a279ae0c0fe74b8ac4d11b38afcaae7"
server_key_version: "encryptionkey@0"
I20260928 12:21:18.135983 3739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:18.153672 3739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:18.156258 3739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:18.156405 3739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:18.157133 3739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:18.157176 3739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:18.157202 3739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:18.157233 3739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:18.164793 3739 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.3:38179
I20260928 12:21:18.165009 3869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.3:38179 every 8 connection(s)
I20260928 12:21:18.165350 3739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
Sep 28 12:21:18 dist-test-slave-920p krb5kdc[3342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598078, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:18.168908 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:18.157755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:51673 (local address 127.2.56.62:37841)
0928 12:21:18.157822 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:18.157824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:18.158171 (+ 347us) server_negotiation.cc:407] Connection header received
0928 12:21:18.159149 (+ 978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:18.159152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:18.159173 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:18.159275 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:18.160337 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.161209 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.162917 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.163235 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.163320 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:18.166357 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:18.166359 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:18.166359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:18.166368 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:18.167723 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:18.168126 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:18.168127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:18.168127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:18.168173 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:18.168347 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:18.168347 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:18.168347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:18.168457 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:18.168505 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:18.168746 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 12:21:18.168832 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 12:21:18.169144 3763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:18.157746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:37841 (local address 127.2.56.3:51673)
0928 12:21:18.158007 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:18.158053 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:18.158781 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:18.159396 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:18.159403 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:18.159854 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:18.160211 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:18.160221 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.161373 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.161374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:18.162816 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:18.162819 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.163329 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.163330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:18.163506 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:18.164270 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:18.164291 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:18.166223 (+ 1932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:18.167804 (+ 1581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:18.167811 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:18.167814 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:18.168068 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:18.168209 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:18.168210 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:18.168210 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:18.168307 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:18.168502 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:18.168506 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:18.168647 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 12:21:18.168733 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 12:21:18.168874 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":157,"threads_started":1}
I20260928 12:21:18.169241 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 3739
I20260928 12:21:18.169358 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance
I20260928 12:21:18.169586 2272 external_mini_cluster.cc:1428] Setting key 2061194abcfed1fe3941e6e692bb17a8
I20260928 12:21:18.169731 3870 heartbeater.cc:347] Connected to a master server at 127.2.56.62:37841
I20260928 12:21:18.169827 3870 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:18.170066 3870 heartbeater.cc:511] Master 127.2.56.62:37841 requested a full tablet report, sending...
I20260928 12:21:18.170495 3383 ts_manager.cc:194] Registered new tserver with Master: 72e5033472634cb88da6f701c6aeb7b3 (127.2.56.3:38179)
I20260928 12:21:18.171029 3383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.3@KRBTEST.COM'} at 127.2.56.3:51673
I20260928 12:21:18.176345 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:18.170869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:37841 (local address 127.2.56.3:42705)
0928 12:21:18.170998 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:18.171009 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:18.171079 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:18.171370 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:18.171371 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:18.171541 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:18.171605 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:18.171607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.172297 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.172298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:18.172833 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:18.172834 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.173177 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.173178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:18.173255 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:18.173725 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:18.173730 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:18.174032 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:18.175412 (+ 1380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:18.175413 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:18.175414 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:18.175575 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:18.175741 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:18.175742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:18.175742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:18.175799 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:18.176013 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:18.176014 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:18.176059 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 12:21:18.176119 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 12:21:18.176227 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
I20260928 12:21:18.176337 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:18.170935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:42705 (local address 127.2.56.62:37841)
0928 12:21:18.170978 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:18.170980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:18.171063 (+ 83us) server_negotiation.cc:407] Connection header received
0928 12:21:18.171149 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:18.171150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:18.171160 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:18.171211 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:18.171675 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.172206 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.172900 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.173109 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.173167 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:18.174109 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:18.174111 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:18.174111 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:18.174118 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:18.175342 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:18.175634 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:18.175635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:18.175636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:18.175691 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:18.175847 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:18.175848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:18.175849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:18.175958 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:18.176015 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:18.176147 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 12:21:18.176235 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 12:21:18.182053 2272 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 12:21:18 dist-test-slave-920p krb5kdc[3342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598078, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 12:21:18 dist-test-slave-920p krb5kdc[3342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598078, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:18.205549 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:18.198353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49404 (local address 127.2.56.62:37841)
0928 12:21:18.198420 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:18.198422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:18.198429 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:21:18.198455 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:18.198456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:18.198468 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:18.198517 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:18.199144 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.199701 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.200563 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.200762 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.200818 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:18.203209 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:18.203211 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:18.203211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:18.203219 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:18.204416 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:18.204757 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:18.204758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:18.204759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:18.204815 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:18.204971 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:18.204972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:18.204972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:18.205104 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:18.205206 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:18.205348 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 12:21:18.205440 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 12:21:18.207216 3383 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:21:18.209174 3383 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:21:18.216325 3889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:18.211963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35568 (local address 127.2.56.3:38179)
0928 12:21:18.212539 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:18.212545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:18.212553 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:21:18.212596 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:18.212598 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:18.212614 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:18.212825 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:18.212892 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.214007 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.215245 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.215839 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.215908 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:18.215986 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:18.216135 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 12:21:18.216191 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":108,"threads_started":1}
I20260928 12:21:18.216320 3887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:18.211834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:38179 (local address 127.0.0.1:35568)
0928 12:21:18.212173 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:18.212186 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:18.212242 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:18.212708 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:18.212710 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:18.212712 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:18.212758 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:18.212759 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.214093 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.214095 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:18.215156 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:18.215158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.215920 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.215921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:18.216021 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:18.216039 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 12:21:18.216109 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 12:21:18.216204 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":42,"thread_start_us":129,"threads_started":1}
I20260928 12:21:18.216325 3888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:18.211827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:36457 (local address 127.0.0.1:33948)
0928 12:21:18.212092 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:18.212150 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:18.212231 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:18.212485 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:18.212488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:18.212491 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:18.212589 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:18.212592 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.214130 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.214132 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:18.215037 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:18.215041 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.216008 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.216009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:18.216071 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:18.216075 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:21:18.216103 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 12:21:18.216191 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":66,"threads_started":1}
I20260928 12:21:18.216581 3891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:18.212069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33948 (local address 127.2.56.1:36457)
0928 12:21:18.212287 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:18.212293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:18.212305 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:21:18.212350 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:18.212353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:18.212369 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:18.212674 (+ 305us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:18.212757 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.214069 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.215122 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.215902 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.216313 (+ 411us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:18.216389 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:18.216417 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 12:21:18.216464 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":87,"threads_started":1}
I20260928 12:21:18.217156 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:18.212313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60700 (local address 127.2.56.2:40655)
0928 12:21:18.212603 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:18.212608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:18.212619 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:21:18.212663 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:18.212665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:18.212682 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:18.212757 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:18.212868 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.213674 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.215231 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.215963 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.216022 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:18.216084 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:18.216367 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 12:21:18.217039 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":96,"threads_started":1}
I20260928 12:21:18.217367 3890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:18.212063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:40655 (local address 127.0.0.1:60700)
0928 12:21:18.212357 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:18.212366 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:18.212424 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:18.212764 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:18.212765 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:18.212767 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:18.212816 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:18.212817 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.213752 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.213754 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:18.215111 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:18.215114 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.216114 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.216115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:18.216174 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:18.216178 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:21:18.216218 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 12:21:18.216263 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":87,"threads_started":1}
I20260928 12:21:18.217326 3804 tablet_service.cc:1467] Processing CreateTablet for tablet 953e2ce1a559476caeb09799f4b2e48a (DEFAULT_TABLE table=test-table [id=5d127e5822364cdfbdfa599183c35867]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:18.217335 3524 tablet_service.cc:1467] Processing CreateTablet for tablet 953e2ce1a559476caeb09799f4b2e48a (DEFAULT_TABLE table=test-table [id=5d127e5822364cdfbdfa599183c35867]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:18.217814 3524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 953e2ce1a559476caeb09799f4b2e48a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:18.217815 3804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 953e2ce1a559476caeb09799f4b2e48a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:18.218145 3664 tablet_service.cc:1467] Processing CreateTablet for tablet 953e2ce1a559476caeb09799f4b2e48a (DEFAULT_TABLE table=test-table [id=5d127e5822364cdfbdfa599183c35867]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:18.218480 3664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 953e2ce1a559476caeb09799f4b2e48a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:18.221884 3894 tablet_bootstrap.cc:492] T 953e2ce1a559476caeb09799f4b2e48a P 72e5033472634cb88da6f701c6aeb7b3: Bootstrap starting.
I20260928 12:21:18.222069 3895 tablet_bootstrap.cc:492] T 953e2ce1a559476caeb09799f4b2e48a P 9248462f96c0451a9b54e19225f3dd0e: Bootstrap starting.
I20260928 12:21:18.222668 3895 tablet_bootstrap.cc:654] T 953e2ce1a559476caeb09799f4b2e48a P 9248462f96c0451a9b54e19225f3dd0e: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:18.222770 3894 tablet_bootstrap.cc:654] T 953e2ce1a559476caeb09799f4b2e48a P 72e5033472634cb88da6f701c6aeb7b3: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:18.223017 3895 log.cc:824] T 953e2ce1a559476caeb09799f4b2e48a P 9248462f96c0451a9b54e19225f3dd0e: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:18.223062 3894 log.cc:824] T 953e2ce1a559476caeb09799f4b2e48a P 72e5033472634cb88da6f701c6aeb7b3: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:18.223322 3893 tablet_bootstrap.cc:492] T 953e2ce1a559476caeb09799f4b2e48a P 604a530966a44b44a5880c3cae2b2115: Bootstrap starting.
I20260928 12:21:18.223841 3895 tablet_bootstrap.cc:492] T 953e2ce1a559476caeb09799f4b2e48a P 9248462f96c0451a9b54e19225f3dd0e: No bootstrap required, opened a new log
I20260928 12:21:18.223930 3895 ts_tablet_manager.cc:1397] T 953e2ce1a559476caeb09799f4b2e48a P 9248462f96c0451a9b54e19225f3dd0e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:21:18.223968 3893 tablet_bootstrap.cc:654] T 953e2ce1a559476caeb09799f4b2e48a P 604a530966a44b44a5880c3cae2b2115: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:18.224289 3893 log.cc:824] T 953e2ce1a559476caeb09799f4b2e48a P 604a530966a44b44a5880c3cae2b2115: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:18.224859 3894 tablet_bootstrap.cc:492] T 953e2ce1a559476caeb09799f4b2e48a P 72e5033472634cb88da6f701c6aeb7b3: No bootstrap required, opened a new log
I20260928 12:21:18.224934 3894 ts_tablet_manager.cc:1397] T 953e2ce1a559476caeb09799f4b2e48a P 72e5033472634cb88da6f701c6aeb7b3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 12:21:18.225592 3893 tablet_bootstrap.cc:492] T 953e2ce1a559476caeb09799f4b2e48a P 604a530966a44b44a5880c3cae2b2115: No bootstrap required, opened a new log
I20260928 12:21:18.225657 3893 ts_tablet_manager.cc:1397] T 953e2ce1a559476caeb09799f4b2e48a P 604a530966a44b44a5880c3cae2b2115: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 12:21:18.226152 3895 raft_consensus.cc:348] T 953e2ce1a559476caeb09799f4b2e48a P 9248462f96c0451a9b54e19225f3dd0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } }
I20260928 12:21:18.226364 3895 raft_consensus.cc:374] T 953e2ce1a559476caeb09799f4b2e48a P 9248462f96c0451a9b54e19225f3dd0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:18.226403 3895 raft_consensus.cc:729] T 953e2ce1a559476caeb09799f4b2e48a P 9248462f96c0451a9b54e19225f3dd0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9248462f96c0451a9b54e19225f3dd0e, State: Initialized, Role: FOLLOWER
I20260928 12:21:18.226590 3895 consensus_queue.cc:260] T 953e2ce1a559476caeb09799f4b2e48a P 9248462f96c0451a9b54e19225f3dd0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } }
I20260928 12:21:18.226917 3895 ts_tablet_manager.cc:1428] T 953e2ce1a559476caeb09799f4b2e48a P 9248462f96c0451a9b54e19225f3dd0e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:21:18.226925 3590 heartbeater.cc:503] Master 127.2.56.62:37841 was elected leader, sending a full tablet report...
I20260928 12:21:18.227079 3894 raft_consensus.cc:348] T 953e2ce1a559476caeb09799f4b2e48a P 72e5033472634cb88da6f701c6aeb7b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } }
I20260928 12:21:18.227202 3894 raft_consensus.cc:374] T 953e2ce1a559476caeb09799f4b2e48a P 72e5033472634cb88da6f701c6aeb7b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:18.227222 3894 raft_consensus.cc:729] T 953e2ce1a559476caeb09799f4b2e48a P 72e5033472634cb88da6f701c6aeb7b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72e5033472634cb88da6f701c6aeb7b3, State: Initialized, Role: FOLLOWER
I20260928 12:21:18.227350 3894 consensus_queue.cc:260] T 953e2ce1a559476caeb09799f4b2e48a P 72e5033472634cb88da6f701c6aeb7b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } }
I20260928 12:21:18.227480 3893 raft_consensus.cc:348] T 953e2ce1a559476caeb09799f4b2e48a P 604a530966a44b44a5880c3cae2b2115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } }
I20260928 12:21:18.227622 3894 ts_tablet_manager.cc:1428] T 953e2ce1a559476caeb09799f4b2e48a P 72e5033472634cb88da6f701c6aeb7b3: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 12:21:18.227641 3893 raft_consensus.cc:374] T 953e2ce1a559476caeb09799f4b2e48a P 604a530966a44b44a5880c3cae2b2115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:18.227676 3893 raft_consensus.cc:729] T 953e2ce1a559476caeb09799f4b2e48a P 604a530966a44b44a5880c3cae2b2115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 604a530966a44b44a5880c3cae2b2115, State: Initialized, Role: FOLLOWER
I20260928 12:21:18.227725 3870 heartbeater.cc:503] Master 127.2.56.62:37841 was elected leader, sending a full tablet report...
I20260928 12:21:18.227787 3893 consensus_queue.cc:260] T 953e2ce1a559476caeb09799f4b2e48a P 604a530966a44b44a5880c3cae2b2115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } }
I20260928 12:21:18.228085 3893 ts_tablet_manager.cc:1428] T 953e2ce1a559476caeb09799f4b2e48a P 604a530966a44b44a5880c3cae2b2115: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 12:21:18.228288 3730 heartbeater.cc:503] Master 127.2.56.62:37841 was elected leader, sending a full tablet report...
W20260928 12:21:18.242801 3731 tablet.cc:2369] T 953e2ce1a559476caeb09799f4b2e48a P 604a530966a44b44a5880c3cae2b2115: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:21:18.328894 3591 tablet.cc:2369] T 953e2ce1a559476caeb09799f4b2e48a P 9248462f96c0451a9b54e19225f3dd0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:21:18.400672 3901 raft_consensus.cc:482] T 953e2ce1a559476caeb09799f4b2e48a P 604a530966a44b44a5880c3cae2b2115 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:21:18.400847 3901 raft_consensus.cc:504] T 953e2ce1a559476caeb09799f4b2e48a P 604a530966a44b44a5880c3cae2b2115 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } }
I20260928 12:21:18.401368 3901 leader_election.cc:288] T 953e2ce1a559476caeb09799f4b2e48a P 604a530966a44b44a5880c3cae2b2115 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9248462f96c0451a9b54e19225f3dd0e (127.2.56.1:36457), 72e5033472634cb88da6f701c6aeb7b3 (127.2.56.3:38179)
I20260928 12:21:18.404426 3891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:18.401643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:48903 (local address 127.2.56.1:36457)
0928 12:21:18.401700 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:18.401703 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:18.401898 (+ 195us) server_negotiation.cc:407] Connection header received
0928 12:21:18.401941 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:18.401942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:18.401957 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:18.402004 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:18.402158 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.402638 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.403699 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.404088 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.404152 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:18.404164 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:18.404299 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 12:21:18.404347 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 12:21:18.404444 3902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:18.401588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:36457 (local address 127.2.56.2:48903)
0928 12:21:18.401820 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:18.401829 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:18.401888 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:18.402014 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:18.402015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:18.402016 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:18.402097 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:18.402098 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.402718 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.402719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:18.403606 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:18.403608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.404163 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.404164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:18.404236 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:18.404240 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:21:18.404285 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 12:21:18.404336 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":87,"thread_start_us":116,"threads_started":1}
I20260928 12:21:18.404572 3623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:18.401588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:38179 (local address 127.2.56.2:43943)
0928 12:21:18.401693 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:18.401706 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:18.401755 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:18.401943 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:18.401944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:18.401947 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:18.402026 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:18.402028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.402720 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.402721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:18.403605 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:18.403608 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.404165 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.404166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:18.404400 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:18.404403 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:21:18.404455 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 12:21:18.404499 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":37}
I20260928 12:21:18.404585 3889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:18.401718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:43943 (local address 127.2.56.3:38179)
0928 12:21:18.401767 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:18.401770 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:18.401782 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:21:18.401819 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:18.401820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:18.401834 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:18.401930 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:18.402094 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.402635 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.403699 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.404090 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.404152 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:18.404164 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:18.404475 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 12:21:18.404522 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 12:21:18.404853 3544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953e2ce1a559476caeb09799f4b2e48a" candidate_uuid: "604a530966a44b44a5880c3cae2b2115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9248462f96c0451a9b54e19225f3dd0e" is_pre_election: true
I20260928 12:21:18.404932 3824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953e2ce1a559476caeb09799f4b2e48a" candidate_uuid: "604a530966a44b44a5880c3cae2b2115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72e5033472634cb88da6f701c6aeb7b3" is_pre_election: true
I20260928 12:21:18.405059 3544 raft_consensus.cc:2454] T 953e2ce1a559476caeb09799f4b2e48a P 9248462f96c0451a9b54e19225f3dd0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 604a530966a44b44a5880c3cae2b2115 in term 0.
I20260928 12:21:18.405103 3824 raft_consensus.cc:2454] T 953e2ce1a559476caeb09799f4b2e48a P 72e5033472634cb88da6f701c6aeb7b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 604a530966a44b44a5880c3cae2b2115 in term 0.
I20260928 12:21:18.405473 3616 leader_election.cc:302] T 953e2ce1a559476caeb09799f4b2e48a P 604a530966a44b44a5880c3cae2b2115 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 604a530966a44b44a5880c3cae2b2115, 72e5033472634cb88da6f701c6aeb7b3; no voters:
I20260928 12:21:18.405637 3901 raft_consensus.cc:2781] T 953e2ce1a559476caeb09799f4b2e48a P 604a530966a44b44a5880c3cae2b2115 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:21:18.405701 3901 raft_consensus.cc:482] T 953e2ce1a559476caeb09799f4b2e48a P 604a530966a44b44a5880c3cae2b2115 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:21:18.405725 3901 raft_consensus.cc:3037] T 953e2ce1a559476caeb09799f4b2e48a P 604a530966a44b44a5880c3cae2b2115 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:18.407213 3901 raft_consensus.cc:504] T 953e2ce1a559476caeb09799f4b2e48a P 604a530966a44b44a5880c3cae2b2115 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } }
I20260928 12:21:18.407373 3901 leader_election.cc:288] T 953e2ce1a559476caeb09799f4b2e48a P 604a530966a44b44a5880c3cae2b2115 [CANDIDATE]: Term 1 election: Requested vote from peers 9248462f96c0451a9b54e19225f3dd0e (127.2.56.1:36457), 72e5033472634cb88da6f701c6aeb7b3 (127.2.56.3:38179)
I20260928 12:21:18.407627 3824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953e2ce1a559476caeb09799f4b2e48a" candidate_uuid: "604a530966a44b44a5880c3cae2b2115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72e5033472634cb88da6f701c6aeb7b3"
I20260928 12:21:18.407658 3544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953e2ce1a559476caeb09799f4b2e48a" candidate_uuid: "604a530966a44b44a5880c3cae2b2115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9248462f96c0451a9b54e19225f3dd0e"
I20260928 12:21:18.407724 3824 raft_consensus.cc:3037] T 953e2ce1a559476caeb09799f4b2e48a P 72e5033472634cb88da6f701c6aeb7b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:18.407759 3544 raft_consensus.cc:3037] T 953e2ce1a559476caeb09799f4b2e48a P 9248462f96c0451a9b54e19225f3dd0e [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:18.408638 3544 raft_consensus.cc:2454] T 953e2ce1a559476caeb09799f4b2e48a P 9248462f96c0451a9b54e19225f3dd0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 604a530966a44b44a5880c3cae2b2115 in term 1.
I20260928 12:21:18.408874 3618 leader_election.cc:302] T 953e2ce1a559476caeb09799f4b2e48a P 604a530966a44b44a5880c3cae2b2115 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 604a530966a44b44a5880c3cae2b2115, 9248462f96c0451a9b54e19225f3dd0e; no voters:
I20260928 12:21:18.408978 3901 raft_consensus.cc:2781] T 953e2ce1a559476caeb09799f4b2e48a P 604a530966a44b44a5880c3cae2b2115 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:21:18.409149 3901 raft_consensus.cc:686] T 953e2ce1a559476caeb09799f4b2e48a P 604a530966a44b44a5880c3cae2b2115 [term 1 LEADER]: Becoming Leader. State: Replica: 604a530966a44b44a5880c3cae2b2115, State: Running, Role: LEADER
I20260928 12:21:18.409159 3824 raft_consensus.cc:2454] T 953e2ce1a559476caeb09799f4b2e48a P 72e5033472634cb88da6f701c6aeb7b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 604a530966a44b44a5880c3cae2b2115 in term 1.
I20260928 12:21:18.409301 3901 consensus_queue.cc:237] T 953e2ce1a559476caeb09799f4b2e48a P 604a530966a44b44a5880c3cae2b2115 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } }
I20260928 12:21:18.410352 3383 catalog_manager.cc:5495] T 953e2ce1a559476caeb09799f4b2e48a P 604a530966a44b44a5880c3cae2b2115 reported cstate change: term changed from 0 to 1, leader changed from <none> to 604a530966a44b44a5880c3cae2b2115 (127.2.56.2). New cstate: current_term: 1 leader_uuid: "604a530966a44b44a5880c3cae2b2115" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } health_report { overall_health: UNKNOWN } } }
W20260928 12:21:18.416103 3871 tablet.cc:2369] T 953e2ce1a559476caeb09799f4b2e48a P 72e5033472634cb88da6f701c6aeb7b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:21:18.444121 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:18.441645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60706 (local address 127.2.56.2:40655)
0928 12:21:18.441711 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:18.441713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:18.441722 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:21:18.441770 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:18.441770 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:18.441793 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:18.441852 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:21:18.441978 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.442492 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.443270 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.443482 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.443536 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:18.443762 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:21:18.443877 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:21:18.443982 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:18.444001 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 12:21:18.444023 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 12:21:18.446931 3544 raft_consensus.cc:1261] T 953e2ce1a559476caeb09799f4b2e48a P 9248462f96c0451a9b54e19225f3dd0e [term 1 FOLLOWER]: Refusing update from remote peer 604a530966a44b44a5880c3cae2b2115: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:21:18.447027 3824 raft_consensus.cc:1261] T 953e2ce1a559476caeb09799f4b2e48a P 72e5033472634cb88da6f701c6aeb7b3 [term 1 FOLLOWER]: Refusing update from remote peer 604a530966a44b44a5880c3cae2b2115: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:21:18.447445 3903 consensus_queue.cc:1035] T 953e2ce1a559476caeb09799f4b2e48a P 604a530966a44b44a5880c3cae2b2115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:21:18.447619 3901 consensus_queue.cc:1035] T 953e2ce1a559476caeb09799f4b2e48a P 604a530966a44b44a5880c3cae2b2115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:21:18.450945 3908 mvcc.cc:205] Tried to move back new op lower bound from 7334289729314934784 to 7334289729165201408. Current Snapshot: MvccSnapshot[applied={T|T < 7334289729314934784}]
I20260928 12:21:18.451635 3907 mvcc.cc:205] Tried to move back new op lower bound from 7334289729314934784 to 7334289729165201408. Current Snapshot: MvccSnapshot[applied={T|T < 7334289729314934784}]
I20260928 12:21:18.456074 3387 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:21:18.456228 3387 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:21:18.457175 3387 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 628ade5983aa468a9f6246ca00163a30: Sending DeleteTablet for 3 replicas of tablet 953e2ce1a559476caeb09799f4b2e48a
I20260928 12:21:18.457684 3664 tablet_service.cc:1514] Processing DeleteTablet for tablet 953e2ce1a559476caeb09799f4b2e48a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:18 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:60700
I20260928 12:21:18.457688 3804 tablet_service.cc:1514] Processing DeleteTablet for tablet 953e2ce1a559476caeb09799f4b2e48a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:18 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:35568
I20260928 12:21:18.457769 3524 tablet_service.cc:1514] Processing DeleteTablet for tablet 953e2ce1a559476caeb09799f4b2e48a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:18 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:33948
I20260928 12:21:18.458067 3387 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:21:18.458258 3387 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:21:18.459095 3918 tablet_replica.cc:330] T 953e2ce1a559476caeb09799f4b2e48a P 72e5033472634cb88da6f701c6aeb7b3: stopping tablet replica
I20260928 12:21:18.459245 3918 raft_consensus.cc:2229] T 953e2ce1a559476caeb09799f4b2e48a P 72e5033472634cb88da6f701c6aeb7b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:21:18.459403 3918 raft_consensus.cc:2258] T 953e2ce1a559476caeb09799f4b2e48a P 72e5033472634cb88da6f701c6aeb7b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:18.458213 3916 tablet_replica.cc:330] T 953e2ce1a559476caeb09799f4b2e48a P 604a530966a44b44a5880c3cae2b2115: stopping tablet replica
I20260928 12:21:18.459949 3918 ts_tablet_manager.cc:1905] T 953e2ce1a559476caeb09799f4b2e48a P 72e5033472634cb88da6f701c6aeb7b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:18.460055 3916 raft_consensus.cc:2229] T 953e2ce1a559476caeb09799f4b2e48a P 604a530966a44b44a5880c3cae2b2115 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:21:18.460342 3916 raft_consensus.cc:2258] T 953e2ce1a559476caeb09799f4b2e48a P 604a530966a44b44a5880c3cae2b2115 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:18.460446 3804 tablet_service.cc:1467] Processing CreateTablet for tablet 451122e542cf444488fd6ca4005aebc3 (DEFAULT_TABLE table=test-table [id=8fabaeace70648c09a556d80fead3445]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:18.460577 3664 tablet_service.cc:1467] Processing CreateTablet for tablet 451122e542cf444488fd6ca4005aebc3 (DEFAULT_TABLE table=test-table [id=8fabaeace70648c09a556d80fead3445]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:18.460718 3664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 451122e542cf444488fd6ca4005aebc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:18.460747 3917 tablet_replica.cc:330] T 953e2ce1a559476caeb09799f4b2e48a P 9248462f96c0451a9b54e19225f3dd0e: stopping tablet replica
I20260928 12:21:18.460892 3917 raft_consensus.cc:2229] T 953e2ce1a559476caeb09799f4b2e48a P 9248462f96c0451a9b54e19225f3dd0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:21:18.461014 3917 raft_consensus.cc:2258] T 953e2ce1a559476caeb09799f4b2e48a P 9248462f96c0451a9b54e19225f3dd0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:18.461037 3916 ts_tablet_manager.cc:1905] T 953e2ce1a559476caeb09799f4b2e48a P 604a530966a44b44a5880c3cae2b2115: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:18.461599 3917 ts_tablet_manager.cc:1905] T 953e2ce1a559476caeb09799f4b2e48a P 9248462f96c0451a9b54e19225f3dd0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:18.462920 3893 tablet_bootstrap.cc:492] T 451122e542cf444488fd6ca4005aebc3 P 604a530966a44b44a5880c3cae2b2115: Bootstrap starting.
I20260928 12:21:18.463079 3524 tablet_service.cc:1467] Processing CreateTablet for tablet 451122e542cf444488fd6ca4005aebc3 (DEFAULT_TABLE table=test-table [id=8fabaeace70648c09a556d80fead3445]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:18.463189 3524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 451122e542cf444488fd6ca4005aebc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:18.463701 3917 ts_tablet_manager.cc:1918] T 953e2ce1a559476caeb09799f4b2e48a P 9248462f96c0451a9b54e19225f3dd0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:21:18.463778 3917 log.cc:1196] T 953e2ce1a559476caeb09799f4b2e48a P 9248462f96c0451a9b54e19225f3dd0e: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-0/wal/wals/953e2ce1a559476caeb09799f4b2e48a
I20260928 12:21:18.464005 3893 tablet_bootstrap.cc:654] T 451122e542cf444488fd6ca4005aebc3 P 604a530966a44b44a5880c3cae2b2115: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:18.464056 3917 ts_tablet_manager.cc:1939] T 953e2ce1a559476caeb09799f4b2e48a P 9248462f96c0451a9b54e19225f3dd0e: Deleting consensus metadata
I20260928 12:21:18.464576 3374 catalog_manager.cc:4841] TS 9248462f96c0451a9b54e19225f3dd0e (127.2.56.1:36457): tablet 953e2ce1a559476caeb09799f4b2e48a (table test-table [id=5d127e5822364cdfbdfa599183c35867]) successfully deleted
I20260928 12:21:18.464931 3895 tablet_bootstrap.cc:492] T 451122e542cf444488fd6ca4005aebc3 P 9248462f96c0451a9b54e19225f3dd0e: Bootstrap starting.
I20260928 12:21:18.465145 3804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 451122e542cf444488fd6ca4005aebc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:18.465704 3895 tablet_bootstrap.cc:654] T 451122e542cf444488fd6ca4005aebc3 P 9248462f96c0451a9b54e19225f3dd0e: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:18.466019 3893 tablet_bootstrap.cc:492] T 451122e542cf444488fd6ca4005aebc3 P 604a530966a44b44a5880c3cae2b2115: No bootstrap required, opened a new log
I20260928 12:21:18.466150 3893 ts_tablet_manager.cc:1397] T 451122e542cf444488fd6ca4005aebc3 P 604a530966a44b44a5880c3cae2b2115: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 12:21:18.466571 3893 raft_consensus.cc:348] T 451122e542cf444488fd6ca4005aebc3 P 604a530966a44b44a5880c3cae2b2115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } }
I20260928 12:21:18.466650 3893 raft_consensus.cc:374] T 451122e542cf444488fd6ca4005aebc3 P 604a530966a44b44a5880c3cae2b2115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:18.466668 3893 raft_consensus.cc:729] T 451122e542cf444488fd6ca4005aebc3 P 604a530966a44b44a5880c3cae2b2115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 604a530966a44b44a5880c3cae2b2115, State: Initialized, Role: FOLLOWER
I20260928 12:21:18.466717 3893 consensus_queue.cc:260] T 451122e542cf444488fd6ca4005aebc3 P 604a530966a44b44a5880c3cae2b2115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } }
I20260928 12:21:18.466810 3893 ts_tablet_manager.cc:1428] T 451122e542cf444488fd6ca4005aebc3 P 604a530966a44b44a5880c3cae2b2115: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:21:18.467334 3894 tablet_bootstrap.cc:492] T 451122e542cf444488fd6ca4005aebc3 P 72e5033472634cb88da6f701c6aeb7b3: Bootstrap starting.
I20260928 12:21:18.467742 3894 tablet_bootstrap.cc:654] T 451122e542cf444488fd6ca4005aebc3 P 72e5033472634cb88da6f701c6aeb7b3: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:18.468533 3894 tablet_bootstrap.cc:492] T 451122e542cf444488fd6ca4005aebc3 P 72e5033472634cb88da6f701c6aeb7b3: No bootstrap required, opened a new log
I20260928 12:21:18.468583 3894 ts_tablet_manager.cc:1397] T 451122e542cf444488fd6ca4005aebc3 P 72e5033472634cb88da6f701c6aeb7b3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 12:21:18.468740 3894 raft_consensus.cc:348] T 451122e542cf444488fd6ca4005aebc3 P 72e5033472634cb88da6f701c6aeb7b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } }
I20260928 12:21:18.468804 3894 raft_consensus.cc:374] T 451122e542cf444488fd6ca4005aebc3 P 72e5033472634cb88da6f701c6aeb7b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:18.468822 3894 raft_consensus.cc:729] T 451122e542cf444488fd6ca4005aebc3 P 72e5033472634cb88da6f701c6aeb7b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72e5033472634cb88da6f701c6aeb7b3, State: Initialized, Role: FOLLOWER
I20260928 12:21:18.468863 3894 consensus_queue.cc:260] T 451122e542cf444488fd6ca4005aebc3 P 72e5033472634cb88da6f701c6aeb7b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } }
I20260928 12:21:18.468962 3894 ts_tablet_manager.cc:1428] T 451122e542cf444488fd6ca4005aebc3 P 72e5033472634cb88da6f701c6aeb7b3: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 12:21:18.469342 3895 tablet_bootstrap.cc:492] T 451122e542cf444488fd6ca4005aebc3 P 9248462f96c0451a9b54e19225f3dd0e: No bootstrap required, opened a new log
I20260928 12:21:18.469400 3895 ts_tablet_manager.cc:1397] T 451122e542cf444488fd6ca4005aebc3 P 9248462f96c0451a9b54e19225f3dd0e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260928 12:21:18.469554 3895 raft_consensus.cc:348] T 451122e542cf444488fd6ca4005aebc3 P 9248462f96c0451a9b54e19225f3dd0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } }
I20260928 12:21:18.469584 3918 ts_tablet_manager.cc:1918] T 953e2ce1a559476caeb09799f4b2e48a P 72e5033472634cb88da6f701c6aeb7b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:21:18.469621 3895 raft_consensus.cc:374] T 451122e542cf444488fd6ca4005aebc3 P 9248462f96c0451a9b54e19225f3dd0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:18.469640 3895 raft_consensus.cc:729] T 451122e542cf444488fd6ca4005aebc3 P 9248462f96c0451a9b54e19225f3dd0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9248462f96c0451a9b54e19225f3dd0e, State: Initialized, Role: FOLLOWER
I20260928 12:21:18.469666 3918 log.cc:1196] T 953e2ce1a559476caeb09799f4b2e48a P 72e5033472634cb88da6f701c6aeb7b3: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-2/wal/wals/953e2ce1a559476caeb09799f4b2e48a
I20260928 12:21:18.469709 3895 consensus_queue.cc:260] T 451122e542cf444488fd6ca4005aebc3 P 9248462f96c0451a9b54e19225f3dd0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } }
I20260928 12:21:18.469952 3916 ts_tablet_manager.cc:1918] T 953e2ce1a559476caeb09799f4b2e48a P 604a530966a44b44a5880c3cae2b2115: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:21:18.470044 3918 ts_tablet_manager.cc:1939] T 953e2ce1a559476caeb09799f4b2e48a P 72e5033472634cb88da6f701c6aeb7b3: Deleting consensus metadata
I20260928 12:21:18.470069 3916 log.cc:1196] T 953e2ce1a559476caeb09799f4b2e48a P 604a530966a44b44a5880c3cae2b2115: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-1/wal/wals/953e2ce1a559476caeb09799f4b2e48a
I20260928 12:21:18.470111 3895 ts_tablet_manager.cc:1428] T 451122e542cf444488fd6ca4005aebc3 P 9248462f96c0451a9b54e19225f3dd0e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 12:21:18.470301 3916 ts_tablet_manager.cc:1939] T 953e2ce1a559476caeb09799f4b2e48a P 604a530966a44b44a5880c3cae2b2115: Deleting consensus metadata
I20260928 12:21:18.470806 3374 catalog_manager.cc:4841] TS 604a530966a44b44a5880c3cae2b2115 (127.2.56.2:40655): tablet 953e2ce1a559476caeb09799f4b2e48a (table test-table [id=5d127e5822364cdfbdfa599183c35867]) successfully deleted
I20260928 12:21:18.470973 3372 catalog_manager.cc:4841] TS 72e5033472634cb88da6f701c6aeb7b3 (127.2.56.3:38179): tablet 953e2ce1a559476caeb09799f4b2e48a (table test-table [id=5d127e5822364cdfbdfa599183c35867]) successfully deleted
I20260928 12:21:18.721346 3901 raft_consensus.cc:482] T 451122e542cf444488fd6ca4005aebc3 P 604a530966a44b44a5880c3cae2b2115 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:21:18.721488 3901 raft_consensus.cc:504] T 451122e542cf444488fd6ca4005aebc3 P 604a530966a44b44a5880c3cae2b2115 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } }
I20260928 12:21:18.721688 3901 leader_election.cc:288] T 451122e542cf444488fd6ca4005aebc3 P 604a530966a44b44a5880c3cae2b2115 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9248462f96c0451a9b54e19225f3dd0e (127.2.56.1:36457), 72e5033472634cb88da6f701c6aeb7b3 (127.2.56.3:38179)
I20260928 12:21:18.721999 3824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "451122e542cf444488fd6ca4005aebc3" candidate_uuid: "604a530966a44b44a5880c3cae2b2115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72e5033472634cb88da6f701c6aeb7b3" is_pre_election: true
I20260928 12:21:18.722015 3544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "451122e542cf444488fd6ca4005aebc3" candidate_uuid: "604a530966a44b44a5880c3cae2b2115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9248462f96c0451a9b54e19225f3dd0e" is_pre_election: true
I20260928 12:21:18.722103 3824 raft_consensus.cc:2454] T 451122e542cf444488fd6ca4005aebc3 P 72e5033472634cb88da6f701c6aeb7b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 604a530966a44b44a5880c3cae2b2115 in term 0.
I20260928 12:21:18.722127 3544 raft_consensus.cc:2454] T 451122e542cf444488fd6ca4005aebc3 P 9248462f96c0451a9b54e19225f3dd0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 604a530966a44b44a5880c3cae2b2115 in term 0.
I20260928 12:21:18.722280 3616 leader_election.cc:302] T 451122e542cf444488fd6ca4005aebc3 P 604a530966a44b44a5880c3cae2b2115 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 604a530966a44b44a5880c3cae2b2115, 72e5033472634cb88da6f701c6aeb7b3; no voters:
I20260928 12:21:18.722396 3901 raft_consensus.cc:2781] T 451122e542cf444488fd6ca4005aebc3 P 604a530966a44b44a5880c3cae2b2115 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:21:18.722438 3901 raft_consensus.cc:482] T 451122e542cf444488fd6ca4005aebc3 P 604a530966a44b44a5880c3cae2b2115 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:21:18.722458 3901 raft_consensus.cc:3037] T 451122e542cf444488fd6ca4005aebc3 P 604a530966a44b44a5880c3cae2b2115 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:18.723352 3901 raft_consensus.cc:504] T 451122e542cf444488fd6ca4005aebc3 P 604a530966a44b44a5880c3cae2b2115 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } }
I20260928 12:21:18.723496 3901 leader_election.cc:288] T 451122e542cf444488fd6ca4005aebc3 P 604a530966a44b44a5880c3cae2b2115 [CANDIDATE]: Term 1 election: Requested vote from peers 9248462f96c0451a9b54e19225f3dd0e (127.2.56.1:36457), 72e5033472634cb88da6f701c6aeb7b3 (127.2.56.3:38179)
I20260928 12:21:18.723670 3824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "451122e542cf444488fd6ca4005aebc3" candidate_uuid: "604a530966a44b44a5880c3cae2b2115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72e5033472634cb88da6f701c6aeb7b3"
I20260928 12:21:18.723750 3824 raft_consensus.cc:3037] T 451122e542cf444488fd6ca4005aebc3 P 72e5033472634cb88da6f701c6aeb7b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:18.723902 3544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "451122e542cf444488fd6ca4005aebc3" candidate_uuid: "604a530966a44b44a5880c3cae2b2115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9248462f96c0451a9b54e19225f3dd0e"
I20260928 12:21:18.723999 3544 raft_consensus.cc:3037] T 451122e542cf444488fd6ca4005aebc3 P 9248462f96c0451a9b54e19225f3dd0e [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:18.724506 3824 raft_consensus.cc:2454] T 451122e542cf444488fd6ca4005aebc3 P 72e5033472634cb88da6f701c6aeb7b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 604a530966a44b44a5880c3cae2b2115 in term 1.
I20260928 12:21:18.724704 3616 leader_election.cc:302] T 451122e542cf444488fd6ca4005aebc3 P 604a530966a44b44a5880c3cae2b2115 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 604a530966a44b44a5880c3cae2b2115, 72e5033472634cb88da6f701c6aeb7b3; no voters:
I20260928 12:21:18.724800 3901 raft_consensus.cc:2781] T 451122e542cf444488fd6ca4005aebc3 P 604a530966a44b44a5880c3cae2b2115 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:21:18.724799 3544 raft_consensus.cc:2454] T 451122e542cf444488fd6ca4005aebc3 P 9248462f96c0451a9b54e19225f3dd0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 604a530966a44b44a5880c3cae2b2115 in term 1.
I20260928 12:21:18.724875 3901 raft_consensus.cc:686] T 451122e542cf444488fd6ca4005aebc3 P 604a530966a44b44a5880c3cae2b2115 [term 1 LEADER]: Becoming Leader. State: Replica: 604a530966a44b44a5880c3cae2b2115, State: Running, Role: LEADER
I20260928 12:21:18.724946 3901 consensus_queue.cc:237] T 451122e542cf444488fd6ca4005aebc3 P 604a530966a44b44a5880c3cae2b2115 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } }
I20260928 12:21:18.725641 3383 catalog_manager.cc:5495] T 451122e542cf444488fd6ca4005aebc3 P 604a530966a44b44a5880c3cae2b2115 reported cstate change: term changed from 0 to 1, leader changed from <none> to 604a530966a44b44a5880c3cae2b2115 (127.2.56.2). New cstate: current_term: 1 leader_uuid: "604a530966a44b44a5880c3cae2b2115" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } health_report { overall_health: UNKNOWN } } }
Sep 28 12:21:18 dist-test-slave-920p krb5kdc[3342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598077, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:18.789227 3929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:18.782283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49412 (local address 127.2.56.62:37841)
0928 12:21:18.782491 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:18.782493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:18.782524 (+ 31us) server_negotiation.cc:407] Connection header received
0928 12:21:18.782549 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:18.782550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:18.782561 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:18.782615 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:18.783165 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.783730 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.784449 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.784656 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.784715 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:18.786919 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:18.786920 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:18.786921 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:18.786928 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:18.788156 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:18.788507 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:18.788508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:18.788508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:18.788562 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:18.788710 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:18.788710 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:18.788711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:18.788820 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:18.788876 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:18.789020 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 12:21:18.789104 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":116,"threads_started":1}
I20260928 12:21:18.789305 3928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:18.782226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:37841 (local address 127.0.0.1:49412)
0928 12:21:18.782394 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:18.782408 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:18.782491 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:18.782628 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:18.782628 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:18.783025 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:18.783096 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:18.783099 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.783785 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.783786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:18.784388 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:18.784390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.784725 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.784726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:18.784778 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:18.785354 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:18.785359 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:18.786836 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:18.788225 (+ 1389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:18.788234 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:18.788237 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:18.788431 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:18.788612 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:18.788612 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:18.788613 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:18.788664 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:18.788869 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:18.788878 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:18.788928 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 12:21:18.788990 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 12:21:18.789169 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":73,"thread_start_us":81,"threads_started":1}
I20260928 12:21:18.791481 3383 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:49412:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 12:21:18.791735 3383 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:21:18.793310 3804 tablet_service.cc:1467] Processing CreateTablet for tablet 182ff5a380de4aa9908274cbee845f1d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=73c3a08ce0cd40af97cccbee13dc063f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 12:21:18.793295 3664 tablet_service.cc:1467] Processing CreateTablet for tablet 182ff5a380de4aa9908274cbee845f1d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=73c3a08ce0cd40af97cccbee13dc063f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 12:21:18.793372 3524 tablet_service.cc:1467] Processing CreateTablet for tablet 182ff5a380de4aa9908274cbee845f1d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=73c3a08ce0cd40af97cccbee13dc063f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 12:21:18.793497 3804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 182ff5a380de4aa9908274cbee845f1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:18.793500 3664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 182ff5a380de4aa9908274cbee845f1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:18.793565 3524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 182ff5a380de4aa9908274cbee845f1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:18.795066 3895 tablet_bootstrap.cc:492] T 182ff5a380de4aa9908274cbee845f1d P 9248462f96c0451a9b54e19225f3dd0e: Bootstrap starting.
I20260928 12:21:18.795157 3893 tablet_bootstrap.cc:492] T 182ff5a380de4aa9908274cbee845f1d P 604a530966a44b44a5880c3cae2b2115: Bootstrap starting.
I20260928 12:21:18.795406 3894 tablet_bootstrap.cc:492] T 182ff5a380de4aa9908274cbee845f1d P 72e5033472634cb88da6f701c6aeb7b3: Bootstrap starting.
I20260928 12:21:18.795560 3893 tablet_bootstrap.cc:654] T 182ff5a380de4aa9908274cbee845f1d P 604a530966a44b44a5880c3cae2b2115: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:18.795800 3894 tablet_bootstrap.cc:654] T 182ff5a380de4aa9908274cbee845f1d P 72e5033472634cb88da6f701c6aeb7b3: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:18.795851 3895 tablet_bootstrap.cc:654] T 182ff5a380de4aa9908274cbee845f1d P 9248462f96c0451a9b54e19225f3dd0e: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:18.796273 3893 tablet_bootstrap.cc:492] T 182ff5a380de4aa9908274cbee845f1d P 604a530966a44b44a5880c3cae2b2115: No bootstrap required, opened a new log
I20260928 12:21:18.796329 3893 ts_tablet_manager.cc:1397] T 182ff5a380de4aa9908274cbee845f1d P 604a530966a44b44a5880c3cae2b2115: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:21:18.796408 3894 tablet_bootstrap.cc:492] T 182ff5a380de4aa9908274cbee845f1d P 72e5033472634cb88da6f701c6aeb7b3: No bootstrap required, opened a new log
I20260928 12:21:18.796442 3894 ts_tablet_manager.cc:1397] T 182ff5a380de4aa9908274cbee845f1d P 72e5033472634cb88da6f701c6aeb7b3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 12:21:18.796487 3893 raft_consensus.cc:348] T 182ff5a380de4aa9908274cbee845f1d P 604a530966a44b44a5880c3cae2b2115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } }
I20260928 12:21:18.796576 3893 raft_consensus.cc:374] T 182ff5a380de4aa9908274cbee845f1d P 604a530966a44b44a5880c3cae2b2115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:18.796599 3893 raft_consensus.cc:729] T 182ff5a380de4aa9908274cbee845f1d P 604a530966a44b44a5880c3cae2b2115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 604a530966a44b44a5880c3cae2b2115, State: Initialized, Role: FOLLOWER
I20260928 12:21:18.796661 3893 consensus_queue.cc:260] T 182ff5a380de4aa9908274cbee845f1d P 604a530966a44b44a5880c3cae2b2115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } }
I20260928 12:21:18.796787 3893 ts_tablet_manager.cc:1428] T 182ff5a380de4aa9908274cbee845f1d P 604a530966a44b44a5880c3cae2b2115: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:18.796861 3894 raft_consensus.cc:348] T 182ff5a380de4aa9908274cbee845f1d P 72e5033472634cb88da6f701c6aeb7b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } }
I20260928 12:21:18.796938 3894 raft_consensus.cc:374] T 182ff5a380de4aa9908274cbee845f1d P 72e5033472634cb88da6f701c6aeb7b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:18.796967 3894 raft_consensus.cc:729] T 182ff5a380de4aa9908274cbee845f1d P 72e5033472634cb88da6f701c6aeb7b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72e5033472634cb88da6f701c6aeb7b3, State: Initialized, Role: FOLLOWER
I20260928 12:21:18.797032 3894 consensus_queue.cc:260] T 182ff5a380de4aa9908274cbee845f1d P 72e5033472634cb88da6f701c6aeb7b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } }
I20260928 12:21:18.796798 3901 tablet_replica.cc:439] T 182ff5a380de4aa9908274cbee845f1d P 604a530966a44b44a5880c3cae2b2115: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } } }
I20260928 12:21:18.797094 3901 tablet_replica.cc:442] T 182ff5a380de4aa9908274cbee845f1d P 604a530966a44b44a5880c3cae2b2115: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:21:18.797137 3894 ts_tablet_manager.cc:1428] T 182ff5a380de4aa9908274cbee845f1d P 72e5033472634cb88da6f701c6aeb7b3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 12:21:18.797250 3900 tablet_replica.cc:439] T 182ff5a380de4aa9908274cbee845f1d P 72e5033472634cb88da6f701c6aeb7b3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } } }
I20260928 12:21:18.797372 3900 tablet_replica.cc:442] T 182ff5a380de4aa9908274cbee845f1d P 72e5033472634cb88da6f701c6aeb7b3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:21:18.797784 3895 tablet_bootstrap.cc:492] T 182ff5a380de4aa9908274cbee845f1d P 9248462f96c0451a9b54e19225f3dd0e: No bootstrap required, opened a new log
I20260928 12:21:18.797838 3895 ts_tablet_manager.cc:1397] T 182ff5a380de4aa9908274cbee845f1d P 9248462f96c0451a9b54e19225f3dd0e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260928 12:21:18.798295 3895 raft_consensus.cc:348] T 182ff5a380de4aa9908274cbee845f1d P 9248462f96c0451a9b54e19225f3dd0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } }
I20260928 12:21:18.798367 3895 raft_consensus.cc:374] T 182ff5a380de4aa9908274cbee845f1d P 9248462f96c0451a9b54e19225f3dd0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:18.798378 3895 raft_consensus.cc:729] T 182ff5a380de4aa9908274cbee845f1d P 9248462f96c0451a9b54e19225f3dd0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9248462f96c0451a9b54e19225f3dd0e, State: Initialized, Role: FOLLOWER
I20260928 12:21:18.798417 3895 consensus_queue.cc:260] T 182ff5a380de4aa9908274cbee845f1d P 9248462f96c0451a9b54e19225f3dd0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } }
I20260928 12:21:18.798548 3895 ts_tablet_manager.cc:1428] T 182ff5a380de4aa9908274cbee845f1d P 9248462f96c0451a9b54e19225f3dd0e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 12:21:18.798637 3899 tablet_replica.cc:439] T 182ff5a380de4aa9908274cbee845f1d P 9248462f96c0451a9b54e19225f3dd0e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } } }
I20260928 12:21:18.798928 3899 tablet_replica.cc:442] T 182ff5a380de4aa9908274cbee845f1d P 9248462f96c0451a9b54e19225f3dd0e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:21:18.897604 3900 raft_consensus.cc:482] T 182ff5a380de4aa9908274cbee845f1d P 72e5033472634cb88da6f701c6aeb7b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:21:18.897744 3900 raft_consensus.cc:504] T 182ff5a380de4aa9908274cbee845f1d P 72e5033472634cb88da6f701c6aeb7b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } }
I20260928 12:21:18.898206 3900 leader_election.cc:288] T 182ff5a380de4aa9908274cbee845f1d P 72e5033472634cb88da6f701c6aeb7b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9248462f96c0451a9b54e19225f3dd0e (127.2.56.1:36457), 604a530966a44b44a5880c3cae2b2115 (127.2.56.2:40655)
I20260928 12:21:18.901778 3933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:18.898435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:36457 (local address 127.2.56.3:41483)
0928 12:21:18.898584 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:18.898596 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:18.898676 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:18.898915 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:18.898916 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:18.898918 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:18.899024 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:18.899028 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.899857 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.899858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:18.900867 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:18.900870 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.901536 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.901537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:18.901604 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:18.901610 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 12:21:18.901656 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 12:21:18.901694 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":87,"threads_started":1}
I20260928 12:21:18.901808 3891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:18.898525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:41483 (local address 127.2.56.1:36457)
0928 12:21:18.898585 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:18.898587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:18.898669 (+ 82us) server_negotiation.cc:407] Connection header received
0928 12:21:18.898784 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:18.898786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:18.898799 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:18.898833 (+ 34us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:18.899106 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.899781 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.900944 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.901438 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.901499 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:18.901513 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:18.901684 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 12:21:18.901733 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 12:21:18.901986 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:18.898726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:55881 (local address 127.2.56.2:40655)
0928 12:21:18.898805 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:18.898808 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:18.898817 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:21:18.898871 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:18.898872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:18.898881 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:18.898968 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:18.899202 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.899720 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.901044 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.901556 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.901621 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:18.901633 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:18.901858 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 12:21:18.901913 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 12:21:18.902014 3934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:18.898639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:40655 (local address 127.2.56.3:55881)
0928 12:21:18.898736 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:18.898746 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:18.898819 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:18.898975 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:18.898976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:18.898978 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:18.899135 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:18.899137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.899785 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.899786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:18.900966 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:18.900969 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.901624 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.901624 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:18.901781 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:18.901786 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:21:18.901840 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 12:21:18.901889 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":55,"threads_started":1}
I20260928 12:21:18.902329 3544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "182ff5a380de4aa9908274cbee845f1d" candidate_uuid: "72e5033472634cb88da6f701c6aeb7b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9248462f96c0451a9b54e19225f3dd0e" is_pre_election: true
I20260928 12:21:18.902323 3684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "182ff5a380de4aa9908274cbee845f1d" candidate_uuid: "72e5033472634cb88da6f701c6aeb7b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "604a530966a44b44a5880c3cae2b2115" is_pre_election: true
I20260928 12:21:18.902443 3544 raft_consensus.cc:2454] T 182ff5a380de4aa9908274cbee845f1d P 9248462f96c0451a9b54e19225f3dd0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72e5033472634cb88da6f701c6aeb7b3 in term 0.
I20260928 12:21:18.902460 3684 raft_consensus.cc:2454] T 182ff5a380de4aa9908274cbee845f1d P 604a530966a44b44a5880c3cae2b2115 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72e5033472634cb88da6f701c6aeb7b3 in term 0.
I20260928 12:21:18.902637 3758 leader_election.cc:302] T 182ff5a380de4aa9908274cbee845f1d P 72e5033472634cb88da6f701c6aeb7b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72e5033472634cb88da6f701c6aeb7b3, 9248462f96c0451a9b54e19225f3dd0e; no voters:
I20260928 12:21:18.902774 3900 raft_consensus.cc:2781] T 182ff5a380de4aa9908274cbee845f1d P 72e5033472634cb88da6f701c6aeb7b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:21:18.902814 3900 raft_consensus.cc:482] T 182ff5a380de4aa9908274cbee845f1d P 72e5033472634cb88da6f701c6aeb7b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:21:18.902840 3900 raft_consensus.cc:3037] T 182ff5a380de4aa9908274cbee845f1d P 72e5033472634cb88da6f701c6aeb7b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:18.904068 3900 raft_consensus.cc:504] T 182ff5a380de4aa9908274cbee845f1d P 72e5033472634cb88da6f701c6aeb7b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } }
I20260928 12:21:18.904197 3900 leader_election.cc:288] T 182ff5a380de4aa9908274cbee845f1d P 72e5033472634cb88da6f701c6aeb7b3 [CANDIDATE]: Term 1 election: Requested vote from peers 9248462f96c0451a9b54e19225f3dd0e (127.2.56.1:36457), 604a530966a44b44a5880c3cae2b2115 (127.2.56.2:40655)
I20260928 12:21:18.904403 3684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "182ff5a380de4aa9908274cbee845f1d" candidate_uuid: "72e5033472634cb88da6f701c6aeb7b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "604a530966a44b44a5880c3cae2b2115"
I20260928 12:21:18.904397 3544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "182ff5a380de4aa9908274cbee845f1d" candidate_uuid: "72e5033472634cb88da6f701c6aeb7b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9248462f96c0451a9b54e19225f3dd0e"
I20260928 12:21:18.904487 3544 raft_consensus.cc:3037] T 182ff5a380de4aa9908274cbee845f1d P 9248462f96c0451a9b54e19225f3dd0e [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:18.904488 3684 raft_consensus.cc:3037] T 182ff5a380de4aa9908274cbee845f1d P 604a530966a44b44a5880c3cae2b2115 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:18.905373 3544 raft_consensus.cc:2454] T 182ff5a380de4aa9908274cbee845f1d P 9248462f96c0451a9b54e19225f3dd0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72e5033472634cb88da6f701c6aeb7b3 in term 1.
I20260928 12:21:18.905498 3684 raft_consensus.cc:2454] T 182ff5a380de4aa9908274cbee845f1d P 604a530966a44b44a5880c3cae2b2115 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72e5033472634cb88da6f701c6aeb7b3 in term 1.
I20260928 12:21:18.905540 3758 leader_election.cc:302] T 182ff5a380de4aa9908274cbee845f1d P 72e5033472634cb88da6f701c6aeb7b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72e5033472634cb88da6f701c6aeb7b3, 9248462f96c0451a9b54e19225f3dd0e; no voters:
I20260928 12:21:18.905627 3900 raft_consensus.cc:2781] T 182ff5a380de4aa9908274cbee845f1d P 72e5033472634cb88da6f701c6aeb7b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:21:18.905759 3900 raft_consensus.cc:686] T 182ff5a380de4aa9908274cbee845f1d P 72e5033472634cb88da6f701c6aeb7b3 [term 1 LEADER]: Becoming Leader. State: Replica: 72e5033472634cb88da6f701c6aeb7b3, State: Running, Role: LEADER
I20260928 12:21:18.905843 3900 consensus_queue.cc:237] T 182ff5a380de4aa9908274cbee845f1d P 72e5033472634cb88da6f701c6aeb7b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } }
I20260928 12:21:18.906229 3935 tablet_replica.cc:439] T 182ff5a380de4aa9908274cbee845f1d P 72e5033472634cb88da6f701c6aeb7b3: TxnStatusTablet state changed. Reason: New leader 72e5033472634cb88da6f701c6aeb7b3. Latest consensus state: current_term: 1 leader_uuid: "72e5033472634cb88da6f701c6aeb7b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } } }
I20260928 12:21:18.906291 3935 tablet_replica.cc:442] T 182ff5a380de4aa9908274cbee845f1d P 72e5033472634cb88da6f701c6aeb7b3: This TxnStatusTablet replica's current role is: LEADER
I20260928 12:21:18.906631 3937 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 12:21:18.906767 3937 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 12:21:18.906672 3383 catalog_manager.cc:5495] T 182ff5a380de4aa9908274cbee845f1d P 72e5033472634cb88da6f701c6aeb7b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 72e5033472634cb88da6f701c6aeb7b3 (127.2.56.3). New cstate: current_term: 1 leader_uuid: "72e5033472634cb88da6f701c6aeb7b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } health_report { overall_health: HEALTHY } } }
I20260928 12:21:18.918500 3928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:18.914604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:38179 (local address 127.0.0.1:35580)
0928 12:21:18.914707 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:18.914746 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:18.914805 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:18.915208 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:18.915209 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:18.915211 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:21:18.915295 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:18.915298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.916285 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.916286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:18.917333 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:18.917337 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.917882 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.917883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:18.917947 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:18.917972 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:21:18.918261 (+ 289us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:21:18.918271 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 12:21:18.918330 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 12:21:18.918407 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 12:21:18.918643 3940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:18.914801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35580 (local address 127.2.56.3:38179)
0928 12:21:18.915022 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:18.915024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:18.915033 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:21:18.915087 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:18.915088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:18.915108 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:18.915180 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:21:18.915366 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.916198 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.917429 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.917802 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.917861 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:18.918073 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:21:18.918200 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:21:18.918257 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:18.918333 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 12:21:18.918354 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":118,"threads_started":1}
I20260928 12:21:18.920125 3544 raft_consensus.cc:1261] T 182ff5a380de4aa9908274cbee845f1d P 9248462f96c0451a9b54e19225f3dd0e [term 1 FOLLOWER]: Refusing update from remote peer 72e5033472634cb88da6f701c6aeb7b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:21:18.920187 3684 raft_consensus.cc:1261] T 182ff5a380de4aa9908274cbee845f1d P 604a530966a44b44a5880c3cae2b2115 [term 1 FOLLOWER]: Refusing update from remote peer 72e5033472634cb88da6f701c6aeb7b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:21:18.920543 3935 consensus_queue.cc:1035] T 182ff5a380de4aa9908274cbee845f1d P 72e5033472634cb88da6f701c6aeb7b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:21:18.920679 3900 consensus_queue.cc:1035] T 182ff5a380de4aa9908274cbee845f1d P 72e5033472634cb88da6f701c6aeb7b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:21:18.921396 3899 tablet_replica.cc:439] T 182ff5a380de4aa9908274cbee845f1d P 9248462f96c0451a9b54e19225f3dd0e: TxnStatusTablet state changed. Reason: New leader 72e5033472634cb88da6f701c6aeb7b3. Latest consensus state: current_term: 1 leader_uuid: "72e5033472634cb88da6f701c6aeb7b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } } }
I20260928 12:21:18.921386 3901 tablet_replica.cc:439] T 182ff5a380de4aa9908274cbee845f1d P 604a530966a44b44a5880c3cae2b2115: TxnStatusTablet state changed. Reason: New leader 72e5033472634cb88da6f701c6aeb7b3. Latest consensus state: current_term: 1 leader_uuid: "72e5033472634cb88da6f701c6aeb7b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } } }
I20260928 12:21:18.921489 3901 tablet_replica.cc:442] T 182ff5a380de4aa9908274cbee845f1d P 604a530966a44b44a5880c3cae2b2115: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:21:18.921489 3899 tablet_replica.cc:442] T 182ff5a380de4aa9908274cbee845f1d P 9248462f96c0451a9b54e19225f3dd0e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:21:18.922385 3900 tablet_replica.cc:439] T 182ff5a380de4aa9908274cbee845f1d P 72e5033472634cb88da6f701c6aeb7b3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "72e5033472634cb88da6f701c6aeb7b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } } }
I20260928 12:21:18.922473 3900 tablet_replica.cc:442] T 182ff5a380de4aa9908274cbee845f1d P 72e5033472634cb88da6f701c6aeb7b3: This TxnStatusTablet replica's current role is: LEADER
I20260928 12:21:18.922873 3906 mvcc.cc:205] Tried to move back new op lower bound from 7334289731254136832 to 7334289731198935040. Current Snapshot: MvccSnapshot[applied={T|T < 7334289731254136832}]
I20260928 12:21:18.922996 3901 tablet_replica.cc:439] T 182ff5a380de4aa9908274cbee845f1d P 604a530966a44b44a5880c3cae2b2115: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "72e5033472634cb88da6f701c6aeb7b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } } }
I20260928 12:21:18.923070 3901 tablet_replica.cc:442] T 182ff5a380de4aa9908274cbee845f1d P 604a530966a44b44a5880c3cae2b2115: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:21:18.923818 3935 tablet_replica.cc:439] T 182ff5a380de4aa9908274cbee845f1d P 72e5033472634cb88da6f701c6aeb7b3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "72e5033472634cb88da6f701c6aeb7b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } } }
I20260928 12:21:18.923911 3935 tablet_replica.cc:442] T 182ff5a380de4aa9908274cbee845f1d P 72e5033472634cb88da6f701c6aeb7b3: This TxnStatusTablet replica's current role is: LEADER
I20260928 12:21:18.924611 3899 tablet_replica.cc:439] T 182ff5a380de4aa9908274cbee845f1d P 9248462f96c0451a9b54e19225f3dd0e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "72e5033472634cb88da6f701c6aeb7b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 } } peers { permanent_uuid: "604a530966a44b44a5880c3cae2b2115" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 40655 } } peers { permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 } } }
I20260928 12:21:18.924721 3899 tablet_replica.cc:442] T 182ff5a380de4aa9908274cbee845f1d P 9248462f96c0451a9b54e19225f3dd0e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:21:18.929816 3945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:18.926720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:38179 (local address 127.2.56.2:51231)
0928 12:21:18.926912 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:18.926922 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:18.926987 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:18.927139 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:18.927140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:18.927142 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:21:18.927204 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:18.927205 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.928157 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.928157 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:18.928863 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:18.928867 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.929390 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.929391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:18.929444 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:18.929489 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:21:18.929654 (+ 165us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:21:18.929655 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 12:21:18.929687 (+ 32us) client_negotiation.cc:240] Negotiation successful
0928 12:21:18.929746 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":79,"threads_started":1}
I20260928 12:21:18.929808 3940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:18.926788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:51231 (local address 127.2.56.3:38179)
0928 12:21:18.926835 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:18.926837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:18.927033 (+ 196us) server_negotiation.cc:407] Connection header received
0928 12:21:18.927065 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:18.927066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:18.927080 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:18.927138 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:21:18.927242 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.928085 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.928943 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.929316 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.929387 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:18.929563 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:21:18.929602 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:21:18.929651 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:18.929691 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 12:21:18.929714 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:21:18.932526 3544 raft_consensus.cc:1261] T 451122e542cf444488fd6ca4005aebc3 P 9248462f96c0451a9b54e19225f3dd0e [term 1 FOLLOWER]: Refusing update from remote peer 604a530966a44b44a5880c3cae2b2115: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:21:18.932518 3824 raft_consensus.cc:1261] T 451122e542cf444488fd6ca4005aebc3 P 72e5033472634cb88da6f701c6aeb7b3 [term 1 FOLLOWER]: Refusing update from remote peer 604a530966a44b44a5880c3cae2b2115: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:21:18.932751 3911 consensus_queue.cc:1035] T 451122e542cf444488fd6ca4005aebc3 P 604a530966a44b44a5880c3cae2b2115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9248462f96c0451a9b54e19225f3dd0e" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:21:18.932888 3901 consensus_queue.cc:1035] T 451122e542cf444488fd6ca4005aebc3 P 604a530966a44b44a5880c3cae2b2115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72e5033472634cb88da6f701c6aeb7b3" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 38179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:21:18.943383 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:18.940029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:35047 (local address 127.2.56.2:40655)
0928 12:21:18.940087 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:18.940090 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:18.940314 (+ 224us) server_negotiation.cc:407] Connection header received
0928 12:21:18.940362 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:18.940363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:18.940378 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:18.940430 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:21:18.940784 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.941401 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.942726 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.942913 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.942966 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:18.943179 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:21:18.943207 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:21:18.943238 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:18.943295 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 12:21:18.943313 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 12:21:18.943471 3950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:18.940001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:40655 (local address 127.2.56.3:35047)
0928 12:21:18.940235 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:18.940246 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:18.940312 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:18.940433 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:18.940434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:18.940436 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:21:18.940684 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:18.940687 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.941546 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.941547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:18.942644 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:18.942647 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.943000 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.943001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:18.943085 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:18.943106 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:21:18.943244 (+ 138us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:21:18.943245 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 12:21:18.943300 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 12:21:18.943346 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":89,"threads_started":1}
I20260928 12:21:18.951785 3387 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:21:18.951915 3387 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:21:18.952615 3387 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 628ade5983aa468a9f6246ca00163a30: Sending DeleteTablet for 3 replicas of tablet 451122e542cf444488fd6ca4005aebc3
I20260928 12:21:18.952910 3524 tablet_service.cc:1514] Processing DeleteTablet for tablet 451122e542cf444488fd6ca4005aebc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:18 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:33948
I20260928 12:21:18.952908 3804 tablet_service.cc:1514] Processing DeleteTablet for tablet 451122e542cf444488fd6ca4005aebc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:18 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:35568
I20260928 12:21:18.952950 3664 tablet_service.cc:1514] Processing DeleteTablet for tablet 451122e542cf444488fd6ca4005aebc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:18 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:60700
I20260928 12:21:18.953035 3917 tablet_replica.cc:330] T 451122e542cf444488fd6ca4005aebc3 P 9248462f96c0451a9b54e19225f3dd0e: stopping tablet replica
I20260928 12:21:18.953042 3918 tablet_replica.cc:330] T 451122e542cf444488fd6ca4005aebc3 P 72e5033472634cb88da6f701c6aeb7b3: stopping tablet replica
I20260928 12:21:18.953099 3918 raft_consensus.cc:2229] T 451122e542cf444488fd6ca4005aebc3 P 72e5033472634cb88da6f701c6aeb7b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:21:18.953100 3917 raft_consensus.cc:2229] T 451122e542cf444488fd6ca4005aebc3 P 9248462f96c0451a9b54e19225f3dd0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:21:18.953132 3918 raft_consensus.cc:2258] T 451122e542cf444488fd6ca4005aebc3 P 72e5033472634cb88da6f701c6aeb7b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:18.953140 3917 raft_consensus.cc:2258] T 451122e542cf444488fd6ca4005aebc3 P 9248462f96c0451a9b54e19225f3dd0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:18.953193 3916 tablet_replica.cc:330] T 451122e542cf444488fd6ca4005aebc3 P 604a530966a44b44a5880c3cae2b2115: stopping tablet replica
I20260928 12:21:18.953248 3916 raft_consensus.cc:2229] T 451122e542cf444488fd6ca4005aebc3 P 604a530966a44b44a5880c3cae2b2115 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:21:18.953413 3916 raft_consensus.cc:2258] T 451122e542cf444488fd6ca4005aebc3 P 604a530966a44b44a5880c3cae2b2115 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:18.953465 3917 ts_tablet_manager.cc:1905] T 451122e542cf444488fd6ca4005aebc3 P 9248462f96c0451a9b54e19225f3dd0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:18.953696 3918 ts_tablet_manager.cc:1905] T 451122e542cf444488fd6ca4005aebc3 P 72e5033472634cb88da6f701c6aeb7b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:18.953752 3916 ts_tablet_manager.cc:1905] T 451122e542cf444488fd6ca4005aebc3 P 604a530966a44b44a5880c3cae2b2115: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:18.955672 3916 ts_tablet_manager.cc:1918] T 451122e542cf444488fd6ca4005aebc3 P 604a530966a44b44a5880c3cae2b2115: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:21:18.955708 3918 ts_tablet_manager.cc:1918] T 451122e542cf444488fd6ca4005aebc3 P 72e5033472634cb88da6f701c6aeb7b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:21:18.955739 3916 log.cc:1196] T 451122e542cf444488fd6ca4005aebc3 P 604a530966a44b44a5880c3cae2b2115: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-1/wal/wals/451122e542cf444488fd6ca4005aebc3
I20260928 12:21:18.955750 3918 log.cc:1196] T 451122e542cf444488fd6ca4005aebc3 P 72e5033472634cb88da6f701c6aeb7b3: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-2/wal/wals/451122e542cf444488fd6ca4005aebc3
I20260928 12:21:18.955946 3918 ts_tablet_manager.cc:1939] T 451122e542cf444488fd6ca4005aebc3 P 72e5033472634cb88da6f701c6aeb7b3: Deleting consensus metadata
I20260928 12:21:18.955946 3916 ts_tablet_manager.cc:1939] T 451122e542cf444488fd6ca4005aebc3 P 604a530966a44b44a5880c3cae2b2115: Deleting consensus metadata
I20260928 12:21:18.956132 3917 ts_tablet_manager.cc:1918] T 451122e542cf444488fd6ca4005aebc3 P 9248462f96c0451a9b54e19225f3dd0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:21:18.956207 3917 log.cc:1196] T 451122e542cf444488fd6ca4005aebc3 P 9248462f96c0451a9b54e19225f3dd0e: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598072638673-2272-0/minicluster-data/ts-0/wal/wals/451122e542cf444488fd6ca4005aebc3
I20260928 12:21:18.956398 3917 ts_tablet_manager.cc:1939] T 451122e542cf444488fd6ca4005aebc3 P 9248462f96c0451a9b54e19225f3dd0e: Deleting consensus metadata
I20260928 12:21:18.956475 3372 catalog_manager.cc:4841] TS 72e5033472634cb88da6f701c6aeb7b3 (127.2.56.3:38179): tablet 451122e542cf444488fd6ca4005aebc3 (table test-table [id=8fabaeace70648c09a556d80fead3445]) successfully deleted
I20260928 12:21:18.956665 3374 catalog_manager.cc:4841] TS 9248462f96c0451a9b54e19225f3dd0e (127.2.56.1:36457): tablet 451122e542cf444488fd6ca4005aebc3 (table test-table [id=8fabaeace70648c09a556d80fead3445]) successfully deleted
I20260928 12:21:18.956807 3374 catalog_manager.cc:4841] TS 604a530966a44b44a5880c3cae2b2115 (127.2.56.2:40655): tablet 451122e542cf444488fd6ca4005aebc3 (table test-table [id=8fabaeace70648c09a556d80fead3445]) successfully deleted
Sep 28 12:21:18 dist-test-slave-920p krb5kdc[3342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598078, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.56.1@KRBTEST.COM
I20260928 12:21:18.965205 3891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:18.956948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33958 (local address 127.2.56.1:36457)
0928 12:21:18.957003 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:18.957005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:18.957173 (+ 168us) server_negotiation.cc:407] Connection header received
0928 12:21:18.957219 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:18.957220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:18.957236 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:18.957358 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:18.957891 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.958690 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.959380 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.959687 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.959753 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:18.962388 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:18.962401 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:18.962401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:18.962409 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:18.964041 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:18.964442 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:18.964445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:18.964446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:18.964525 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:18.964698 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:18.964699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:18.964699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:18.964849 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:18.964992 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:18.965059 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 12:21:18.965128 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260928 12:21:18.965602 3584 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33958
I20260928 12:21:18.973613 3929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:18.966781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49422 (local address 127.2.56.62:37841)
0928 12:21:18.966833 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:18.966835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:18.966890 (+ 55us) server_negotiation.cc:407] Connection header received
0928 12:21:18.966987 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:18.966988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:18.967001 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:18.967044 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:18.967639 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.968202 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.969239 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:18.969528 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:18.969608 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:18.971087 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:18.971089 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:18.971089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:18.971096 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:18.972435 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:18.972809 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:18.972810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:18.972810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:18.972896 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:18.973069 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:18.973069 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:18.973069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:18.973192 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:18.973312 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:18.973416 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 12:21:18.973513 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260928 12:21:18.973928 3387 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49422
I20260928 12:21:18.974589 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 3459
I20260928 12:21:18.982489 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 3599
I20260928 12:21:18.991497 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 3739
I20260928 12:21:19.000332 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 3355
2026-09-28T12:21:19Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2592 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:21:19 dist-test-slave-920p krb5kdc[3958](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:21:19 dist-test-slave-920p krb5kdc[3958](info): set up 2 sockets
Sep 28 12:21:19 dist-test-slave-920p krb5kdc[3958](info): commencing operation
krb5kdc: starting...
W20260928 12:21:19.961364 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:21:20 dist-test-slave-920p krb5kdc[3958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:21:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:21:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
I20260928 12:21:20.083513 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:39635
--webserver_interface=127.2.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:33209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:39635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:20.176268 3971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:20.176525 3971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:20.176549 3971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:20.177742 3971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:21:20.177793 3971 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 12:21:20.177807 3971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:20.177820 3971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:20.177841 3971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:21:20.177855 3971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:21:20.179346 3971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:33209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:39635
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:39635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:20.179580 3971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:20.179793 3971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:21:20.185343 3971 server_base.cc:1034] running on GCE node
W20260928 12:21:20.185240 3976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:20.185246 3977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:20.185240 3979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:20.185791 3971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:20.186107 3971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:20.187258 3971 hybrid_clock.cc:648] HybridClock initialized: now 1790598080187243 us; error 32 us; skew 500 ppm
Sep 28 12:21:20 dist-test-slave-920p krb5kdc[3958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598080, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:20.208611 3971 init.cc:377] Logged in from keytab as kudu/127.2.56.62@KRBTEST.COM (short username kudu)
I20260928 12:21:20.209285 3971 webserver.cc:460] Webserver started at http://127.2.56.62:39529/ using document root <none> and password file <none>
I20260928 12:21:20.209578 3971 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:20.209635 3971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:20.209780 3971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:20.211059 3971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "4796d476b91e4f5998165aec0ca20110"
format_stamp: "Formatted at 2026-09-28 12:21:20 on dist-test-slave-920p"
server_key: "2ba21f39b9ad7b918cf92028e645a2e2"
server_key_iv: "7cc6385537f6ed92be64a7da52149dca"
server_key_version: "encryptionkey@0"
I20260928 12:21:20.211463 3971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "4796d476b91e4f5998165aec0ca20110"
format_stamp: "Formatted at 2026-09-28 12:21:20 on dist-test-slave-920p"
server_key: "2ba21f39b9ad7b918cf92028e645a2e2"
server_key_iv: "7cc6385537f6ed92be64a7da52149dca"
server_key_version: "encryptionkey@0"
I20260928 12:21:20.212937 3971 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:21:20.213752 3986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:20.213917 3971 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:20.214013 3971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "4796d476b91e4f5998165aec0ca20110"
format_stamp: "Formatted at 2026-09-28 12:21:20 on dist-test-slave-920p"
server_key: "2ba21f39b9ad7b918cf92028e645a2e2"
server_key_iv: "7cc6385537f6ed92be64a7da52149dca"
server_key_version: "encryptionkey@0"
I20260928 12:21:20.214082 3971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:20.229172 3971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:20.232151 3971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:20.232362 3971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:20.237993 3971 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:39635
I20260928 12:21:20.238044 4048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:39635 every 8 connection(s)
I20260928 12:21:20.238440 3971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:21:20.239159 4049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:20.241658 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 3971
I20260928 12:21:20.241762 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:21:20.241963 2272 external_mini_cluster.cc:1428] Setting key 01883513938751bba6d30a02cc6f88c8
I20260928 12:21:20.242149 4049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4796d476b91e4f5998165aec0ca20110: Bootstrap starting.
I20260928 12:21:20.242908 4049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4796d476b91e4f5998165aec0ca20110: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:20.243206 4049 log.cc:824] T 00000000000000000000000000000000 P 4796d476b91e4f5998165aec0ca20110: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:20.243950 4049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4796d476b91e4f5998165aec0ca20110: No bootstrap required, opened a new log
I20260928 12:21:20.246307 4049 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4796d476b91e4f5998165aec0ca20110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4796d476b91e4f5998165aec0ca20110" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39635 } }
I20260928 12:21:20.246582 4049 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4796d476b91e4f5998165aec0ca20110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:20.246650 4049 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4796d476b91e4f5998165aec0ca20110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4796d476b91e4f5998165aec0ca20110, State: Initialized, Role: FOLLOWER
I20260928 12:21:20.246842 4049 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4796d476b91e4f5998165aec0ca20110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4796d476b91e4f5998165aec0ca20110" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39635 } }
I20260928 12:21:20.246932 4049 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4796d476b91e4f5998165aec0ca20110 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:21:20.246973 4049 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4796d476b91e4f5998165aec0ca20110 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:21:20.247017 4049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4796d476b91e4f5998165aec0ca20110 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:20.247679 4049 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4796d476b91e4f5998165aec0ca20110 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4796d476b91e4f5998165aec0ca20110" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39635 } }
I20260928 12:21:20.247794 4049 leader_election.cc:302] T 00000000000000000000000000000000 P 4796d476b91e4f5998165aec0ca20110 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4796d476b91e4f5998165aec0ca20110; no voters:
I20260928 12:21:20.247973 4049 leader_election.cc:288] T 00000000000000000000000000000000 P 4796d476b91e4f5998165aec0ca20110 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:21:20.248011 4054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4796d476b91e4f5998165aec0ca20110 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:21:20.248129 4054 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4796d476b91e4f5998165aec0ca20110 [term 1 LEADER]: Becoming Leader. State: Replica: 4796d476b91e4f5998165aec0ca20110, State: Running, Role: LEADER
I20260928 12:21:20.248219 4054 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4796d476b91e4f5998165aec0ca20110 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4796d476b91e4f5998165aec0ca20110" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39635 } }
I20260928 12:21:20.248282 4049 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4796d476b91e4f5998165aec0ca20110 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:21:20.248654 4055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4796d476b91e4f5998165aec0ca20110 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4796d476b91e4f5998165aec0ca20110" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4796d476b91e4f5998165aec0ca20110" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39635 } } }
I20260928 12:21:20.248759 4055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4796d476b91e4f5998165aec0ca20110 [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:20.248651 4054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4796d476b91e4f5998165aec0ca20110 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4796d476b91e4f5998165aec0ca20110. Latest consensus state: current_term: 1 leader_uuid: "4796d476b91e4f5998165aec0ca20110" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4796d476b91e4f5998165aec0ca20110" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39635 } } }
I20260928 12:21:20.248939 4054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4796d476b91e4f5998165aec0ca20110 [sys.catalog]: This master's current role is: LEADER
Sep 28 12:21:20 dist-test-slave-920p krb5kdc[3958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
W20260928 12:21:20.250244 4070 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4796d476b91e4f5998165aec0ca20110: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:21:20.250317 4070 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:21:20.250363 4060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:21:20.250555 4060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:21:20.252358 4060 catalog_manager.cc:1347] Generated new cluster ID: 76301c604f8f4c579553625007af5708
I20260928 12:21:20.252415 4060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:21:20.253666 4052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:20.242611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48612 (local address 127.2.56.62:39635)
0928 12:21:20.242820 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:20.242830 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:20.242858 (+ 28us) server_negotiation.cc:407] Connection header received
0928 12:21:20.243418 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:20.243427 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:20.243490 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:20.243648 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:20.244440 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.245563 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.246369 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.246912 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.247020 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:20.249980 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:20.249986 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:20.249987 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:20.250003 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:20.251963 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:20.252325 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:20.252329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:20.252330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:20.252430 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:20.252652 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:20.252653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:20.252654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:20.252947 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:20.253097 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:20.253225 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 12:21:20.253469 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":96,"threads_started":1}
I20260928 12:21:20.267944 4060 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:21:20.268543 4060 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:21:20.274609 4060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4796d476b91e4f5998165aec0ca20110: Generated new TSK 0
I20260928 12:21:20.274806 4060 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab.
Entry for principal kudu/127.2.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab.
I20260928 12:21:20.304757 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.1:0
--local_ip_for_outbound_sockets=127.2.56.1
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:39635
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:33209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:20.428429 4075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:20.428707 4075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:20.428747 4075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:20.430162 4075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:20.430214 4075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:20.430243 4075 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 12:21:20.430258 4075 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 12:21:20.430292 4075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.1
I20260928 12:21:20.431931 4075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:33209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.56.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.56.62:39635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.1
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:20.432211 4075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:20.432472 4075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:21:20.438165 4075 server_base.cc:1034] running on GCE node
W20260928 12:21:20.438097 4083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:20.438108 4081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:20.438113 4080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:20.438520 4075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:20.438786 4075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:20.439949 4075 hybrid_clock.cc:648] HybridClock initialized: now 1790598080439934 us; error 31 us; skew 500 ppm
Sep 28 12:21:20 dist-test-slave-920p krb5kdc[3958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598080, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:20.442732 4075 init.cc:377] Logged in from keytab as kudu/127.2.56.1@KRBTEST.COM (short username kudu)
I20260928 12:21:20.443219 4075 webserver.cc:460] Webserver started at http://127.2.56.1:40255/ using document root <none> and password file <none>
I20260928 12:21:20.443445 4075 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:20.443496 4075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:20.443634 4075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:20.444813 4075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-0/data/instance:
uuid: "dd4cf280ae1748ea8b67e9f08551e3bb"
format_stamp: "Formatted at 2026-09-28 12:21:20 on dist-test-slave-920p"
server_key: "026cde8560d139250d154d1ff827e781"
server_key_iv: "297c94d3ba49e1916162190450d70a47"
server_key_version: "encryptionkey@0"
I20260928 12:21:20.445176 4075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance:
uuid: "dd4cf280ae1748ea8b67e9f08551e3bb"
format_stamp: "Formatted at 2026-09-28 12:21:20 on dist-test-slave-920p"
server_key: "026cde8560d139250d154d1ff827e781"
server_key_iv: "297c94d3ba49e1916162190450d70a47"
server_key_version: "encryptionkey@0"
I20260928 12:21:20.446599 4075 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:21:20.447321 4090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:20.447500 4075 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:21:20.447594 4075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-0/wal
uuid: "dd4cf280ae1748ea8b67e9f08551e3bb"
format_stamp: "Formatted at 2026-09-28 12:21:20 on dist-test-slave-920p"
server_key: "026cde8560d139250d154d1ff827e781"
server_key_iv: "297c94d3ba49e1916162190450d70a47"
server_key_version: "encryptionkey@0"
I20260928 12:21:20.447659 4075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:20.464285 4075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:20.466655 4075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:20.466816 4075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:20.467445 4075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:20.467495 4075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:20.467528 4075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:20.467545 4075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:20.475129 4075 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.1:36185
Sep 28 12:21:20 dist-test-slave-920p krb5kdc[3958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598080, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:20.475250 4205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.1:36185 every 8 connection(s)
I20260928 12:21:20.475612 4075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
I20260928 12:21:20.478318 4052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:20.467928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:59233 (local address 127.2.56.62:39635)
0928 12:21:20.467996 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:20.467998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:20.468247 (+ 249us) server_negotiation.cc:407] Connection header received
0928 12:21:20.468830 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:20.468831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:20.468845 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:20.468931 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:20.469883 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.470740 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.472410 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.472769 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.472854 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:20.475744 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:20.475746 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:20.475747 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:20.475755 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:20.477057 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:20.477498 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:20.477499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:20.477499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:20.477555 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:20.477739 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:20.477739 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:20.477739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:20.477849 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:20.477894 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:20.478159 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 12:21:20.478243 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 12:21:20.478477 4099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:20.467926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39635 (local address 127.2.56.1:59233)
0928 12:21:20.468096 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:20.468143 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:20.468725 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:20.469009 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:20.469014 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:20.469416 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:20.469759 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:20.469769 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.470872 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.470873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:20.472299 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:20.472302 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.472873 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.472874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:20.473026 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:20.473617 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:20.473627 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:20.475639 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:20.477138 (+ 1499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:20.477144 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:20.477146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:20.477435 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:20.477606 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:20.477607 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:20.477607 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:20.477704 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:20.477912 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:20.477916 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:20.478049 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 12:21:20.478130 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 12:21:20.478273 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":99,"threads_started":1}
I20260928 12:21:20.479010 4206 heartbeater.cc:347] Connected to a master server at 127.2.56.62:39635
I20260928 12:21:20.479094 4206 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:20.479293 4206 heartbeater.cc:511] Master 127.2.56.62:39635 requested a full tablet report, sending...
I20260928 12:21:20.479872 4002 ts_manager.cc:194] Registered new tserver with Master: dd4cf280ae1748ea8b67e9f08551e3bb (127.2.56.1:36185)
I20260928 12:21:20.480736 4002 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.1@KRBTEST.COM'} at 127.2.56.1:59233
I20260928 12:21:20.481309 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 4075
I20260928 12:21:20.481431 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance
I20260928 12:21:20.481627 2272 external_mini_cluster.cc:1428] Setting key 2846f4af4afb130f273f6735d20dcdab
I20260928 12:21:20.486159 4212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:20.480099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39635 (local address 127.2.56.1:58307)
0928 12:21:20.480271 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:20.480281 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:20.480335 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:20.480446 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:20.480447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:20.480597 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:20.480661 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:20.480662 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.481525 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.481526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:20.482270 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:20.482273 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.482712 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.482712 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:20.482790 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:20.483202 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:20.483207 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:20.483605 (+ 398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:20.485244 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:20.485245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:20.485245 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:20.485447 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:20.485630 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:20.485631 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:20.485631 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:20.485677 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:20.485891 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:20.485892 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:20.485927 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 12:21:20.485991 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 12:21:20.486075 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":90,"threads_started":1}
I20260928 12:21:20.486186 4052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:20.480170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:58307 (local address 127.2.56.62:39635)
0928 12:21:20.480216 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:20.480218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:20.480338 (+ 120us) server_negotiation.cc:407] Connection header received
0928 12:21:20.480381 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:20.480382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:20.480392 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:20.480443 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:20.480718 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.481409 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.482400 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.482639 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.482691 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:20.483678 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:20.483679 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:20.483679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:20.483684 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:20.485161 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:20.485509 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:20.485510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:20.485511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:20.485572 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:20.485720 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:20.485720 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:20.485721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:20.485843 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:20.485904 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:20.486027 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 12:21:20.486102 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
WARNING: no policy specified for kudu/127.2.56.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab.
Entry for principal kudu/127.2.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab.
I20260928 12:21:20.508728 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.2:0
--local_ip_for_outbound_sockets=127.2.56.2
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:39635
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:33209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:20.597859 4215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:20.598146 4215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:20.598187 4215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:20.599507 4215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:20.599556 4215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:20.599586 4215 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 12:21:20.599601 4215 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 12:21:20.599627 4215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.2
I20260928 12:21:20.601332 4215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:33209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.56.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.56.62:39635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.2
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:20.601631 4215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:20.601923 4215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:20.608039 4221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:20.608093 4220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:20.608062 4215 server_base.cc:1034] running on GCE node
W20260928 12:21:20.608031 4223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:20.608433 4215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:20.608734 4215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:20.609920 4215 hybrid_clock.cc:648] HybridClock initialized: now 1790598080609898 us; error 39 us; skew 500 ppm
Sep 28 12:21:20 dist-test-slave-920p krb5kdc[3958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598080, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:20.613071 4215 init.cc:377] Logged in from keytab as kudu/127.2.56.2@KRBTEST.COM (short username kudu)
I20260928 12:21:20.613662 4215 webserver.cc:460] Webserver started at http://127.2.56.2:40205/ using document root <none> and password file <none>
I20260928 12:21:20.613899 4215 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:20.613961 4215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:20.614101 4215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:20.615331 4215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-1/data/instance:
uuid: "a02cd150a0444b2a931e91e89cfa49a2"
format_stamp: "Formatted at 2026-09-28 12:21:20 on dist-test-slave-920p"
server_key: "3211ac74ca8d390d8b9acd5303e99929"
server_key_iv: "512c3a5b400314a2ea997ebe2f5ec9f9"
server_key_version: "encryptionkey@0"
I20260928 12:21:20.615712 4215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance:
uuid: "a02cd150a0444b2a931e91e89cfa49a2"
format_stamp: "Formatted at 2026-09-28 12:21:20 on dist-test-slave-920p"
server_key: "3211ac74ca8d390d8b9acd5303e99929"
server_key_iv: "512c3a5b400314a2ea997ebe2f5ec9f9"
server_key_version: "encryptionkey@0"
I20260928 12:21:20.617244 4215 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:21:20.618113 4230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:20.618255 4215 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:20.618347 4215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-1/wal
uuid: "a02cd150a0444b2a931e91e89cfa49a2"
format_stamp: "Formatted at 2026-09-28 12:21:20 on dist-test-slave-920p"
server_key: "3211ac74ca8d390d8b9acd5303e99929"
server_key_iv: "512c3a5b400314a2ea997ebe2f5ec9f9"
server_key_version: "encryptionkey@0"
I20260928 12:21:20.618412 4215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:20.633539 4215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:20.636001 4215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:20.636152 4215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:20.636806 4215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:20.636847 4215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:20.636880 4215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:20.636904 4215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 12:21:20 dist-test-slave-920p krb5kdc[3958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598080, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:20.645033 4215 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.2:46837
I20260928 12:21:20.645154 4345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.2:46837 every 8 connection(s)
I20260928 12:21:20.645500 4215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
I20260928 12:21:20.647941 4052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:20.637243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:39375 (local address 127.2.56.62:39635)
0928 12:21:20.637326 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:20.637329 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:20.637611 (+ 282us) server_negotiation.cc:407] Connection header received
0928 12:21:20.638416 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:20.638417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:20.638430 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:20.638484 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:20.639564 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.640157 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.641849 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.642101 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.642167 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:20.645184 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:20.645186 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:20.645186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:20.645195 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:20.646674 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:20.647077 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:20.647078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:20.647079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:20.647125 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:20.647315 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:20.647316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:20.647316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:20.647429 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:20.647493 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:20.647786 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 12:21:20.647868 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 12:21:20.648115 4239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:20.637225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39635 (local address 127.2.56.2:39375)
0928 12:21:20.637446 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:20.637503 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:20.638302 (+ 799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:20.638619 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:20.638626 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:20.639040 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:20.639443 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:20.639454 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.640236 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.640238 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:20.641756 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:20.641760 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.642225 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.642226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:20.642424 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:20.643028 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:20.643039 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:20.645029 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:20.646753 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:20.646759 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:20.646761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:20.647014 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:20.647161 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:20.647162 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:20.647162 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:20.647253 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:20.647488 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:20.647492 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:20.647645 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 12:21:20.647759 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 12:21:20.647905 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":121,"threads_started":1}
I20260928 12:21:20.648590 4346 heartbeater.cc:347] Connected to a master server at 127.2.56.62:39635
I20260928 12:21:20.648732 4346 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:20.649003 4346 heartbeater.cc:511] Master 127.2.56.62:39635 requested a full tablet report, sending...
I20260928 12:21:20.649487 4003 ts_manager.cc:194] Registered new tserver with Master: a02cd150a0444b2a931e91e89cfa49a2 (127.2.56.2:46837)
I20260928 12:21:20.649991 4003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.2@KRBTEST.COM'} at 127.2.56.2:39375
I20260928 12:21:20.654362 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 4215
I20260928 12:21:20.654467 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance
I20260928 12:21:20.654659 2272 external_mini_cluster.cc:1428] Setting key 183b865ee0a71327a1b0e77929c3b303
I20260928 12:21:20.655105 4052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:20.649756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:50975 (local address 127.2.56.62:39635)
0928 12:21:20.649795 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:20.649797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:20.649920 (+ 123us) server_negotiation.cc:407] Connection header received
0928 12:21:20.649974 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:20.649974 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:20.649985 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:20.650027 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:20.650344 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.650919 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.651615 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.651829 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.651903 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:20.652759 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:20.652759 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:20.652760 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:20.652764 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:20.653996 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:20.654301 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:20.654303 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:20.654303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:20.654359 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:20.654554 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:20.654555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:20.654555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:20.654715 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:20.654759 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:20.654929 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 12:21:20.655008 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 12:21:20.655119 4352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:20.649711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39635 (local address 127.2.56.2:50975)
0928 12:21:20.649856 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:20.649865 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:20.649926 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:20.650040 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:20.650041 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:20.650211 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:20.650279 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:20.650281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.650996 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.650997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:20.651519 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:20.651521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.651903 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.651904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:20.651967 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:20.652384 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:20.652388 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:20.652697 (+ 309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:20.654060 (+ 1363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:20.654061 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:20.654061 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:20.654233 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:20.654421 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:20.654421 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:20.654422 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:20.654486 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:20.654803 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:20.654804 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:20.654840 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 12:21:20.654892 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 12:21:20.655008 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for kudu/127.2.56.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab.
Entry for principal kudu/127.2.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab.
I20260928 12:21:20.682670 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.3:0
--local_ip_for_outbound_sockets=127.2.56.3
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:39635
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:33209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:20.773830 4355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:20.774117 4355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:20.774158 4355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:20.775441 4355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:20.775486 4355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:20.775516 4355 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 12:21:20.775529 4355 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 12:21:20.775554 4355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.3
I20260928 12:21:20.777122 4355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:33209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.56.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.56.62:39635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.3
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:20.777415 4355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:20.777661 4355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:20.782881 4363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:20.782882 4361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:20.782882 4360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:20.783008 4355 server_base.cc:1034] running on GCE node
I20260928 12:21:20.783269 4355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:20.783509 4355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:20.784642 4355 hybrid_clock.cc:648] HybridClock initialized: now 1790598080784628 us; error 29 us; skew 500 ppm
Sep 28 12:21:20 dist-test-slave-920p krb5kdc[3958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598080, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:20.787300 4355 init.cc:377] Logged in from keytab as kudu/127.2.56.3@KRBTEST.COM (short username kudu)
I20260928 12:21:20.787751 4355 webserver.cc:460] Webserver started at http://127.2.56.3:42271/ using document root <none> and password file <none>
I20260928 12:21:20.787948 4355 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:20.787992 4355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:20.788089 4355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:20.789196 4355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-2/data/instance:
uuid: "646a0acf9a8841009d4a3fe5b5e43150"
format_stamp: "Formatted at 2026-09-28 12:21:20 on dist-test-slave-920p"
server_key: "24fa5707a147fcb875877db912c86c06"
server_key_iv: "54b731ef9bad1025b94722f014f89e4b"
server_key_version: "encryptionkey@0"
I20260928 12:21:20.789587 4355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance:
uuid: "646a0acf9a8841009d4a3fe5b5e43150"
format_stamp: "Formatted at 2026-09-28 12:21:20 on dist-test-slave-920p"
server_key: "24fa5707a147fcb875877db912c86c06"
server_key_iv: "54b731ef9bad1025b94722f014f89e4b"
server_key_version: "encryptionkey@0"
I20260928 12:21:20.790907 4355 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:21:20.791843 4370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:20.792037 4355 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:21:20.792129 4355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-2/wal
uuid: "646a0acf9a8841009d4a3fe5b5e43150"
format_stamp: "Formatted at 2026-09-28 12:21:20 on dist-test-slave-920p"
server_key: "24fa5707a147fcb875877db912c86c06"
server_key_iv: "54b731ef9bad1025b94722f014f89e4b"
server_key_version: "encryptionkey@0"
I20260928 12:21:20.792197 4355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:20.802052 4355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:20.804082 4355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:20.804193 4355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:20.804700 4355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:20.804749 4355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:20.804774 4355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:20.804788 4355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:20.811949 4355 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.3:34425
I20260928 12:21:20.812093 4485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.3:34425 every 8 connection(s)
Sep 28 12:21:20 dist-test-slave-920p krb5kdc[3958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598080, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:20.812378 4355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
I20260928 12:21:20.815017 4052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:20.805176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:51033 (local address 127.2.56.62:39635)
0928 12:21:20.805248 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:20.805250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:20.805472 (+ 222us) server_negotiation.cc:407] Connection header received
0928 12:21:20.806058 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:20.806060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:20.806075 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:20.806133 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:20.807156 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.807997 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.809527 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.809833 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.809921 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:20.812653 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:20.812655 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:20.812655 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:20.812665 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:20.813919 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:20.814273 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:20.814274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:20.814274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:20.814322 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:20.814504 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:20.814504 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:20.814504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:20.814612 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:20.814659 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:20.814873 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 12:21:20.814943 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 12:21:20.815217 4379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:20.805154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39635 (local address 127.2.56.3:51033)
0928 12:21:20.805330 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:20.805374 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:20.805929 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:20.806264 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:20.806270 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:20.806676 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:20.807042 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:20.807051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.808076 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.808078 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:20.809438 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:20.809441 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.809923 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.809924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:20.810100 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:20.810695 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:20.810706 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:20.812530 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:20.813994 (+ 1464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:20.814000 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:20.814002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:20.814213 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:20.814366 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:20.814366 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:20.814366 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:20.814462 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:20.814653 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:20.814657 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:20.814779 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 12:21:20.814860 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 12:21:20.815009 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":81,"threads_started":1}
I20260928 12:21:20.815686 4486 heartbeater.cc:347] Connected to a master server at 127.2.56.62:39635
I20260928 12:21:20.815768 4486 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:20.815979 4486 heartbeater.cc:511] Master 127.2.56.62:39635 requested a full tablet report, sending...
I20260928 12:21:20.816417 4003 ts_manager.cc:194] Registered new tserver with Master: 646a0acf9a8841009d4a3fe5b5e43150 (127.2.56.3:34425)
I20260928 12:21:20.816840 4003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.3@KRBTEST.COM'} at 127.2.56.3:51033
I20260928 12:21:20.818293 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 4355
I20260928 12:21:20.818380 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance
I20260928 12:21:20.818593 2272 external_mini_cluster.cc:1428] Setting key 0ed07d2d8b6dd6925fad579338e2462c
I20260928 12:21:20.819326 2272 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:21:20.822252 4492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:20.816787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39635 (local address 127.2.56.3:60319)
0928 12:21:20.816912 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:20.816922 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:20.817008 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:20.817175 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:20.817176 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:20.817413 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:20.817485 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:20.817488 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.818212 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.818213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:20.818822 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:20.818824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.819206 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.819207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:20.819287 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:20.819678 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:20.819683 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:20.819997 (+ 314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:20.821260 (+ 1263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:20.821284 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:20.821285 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:20.821442 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:20.821603 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:20.821603 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:20.821604 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:20.821660 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:20.821919 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:20.821920 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:20.821971 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 12:21:20.822040 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 12:21:20.822142 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":77,"threads_started":1}
I20260928 12:21:20.822239 4052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:20.816877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:60319 (local address 127.2.56.62:39635)
0928 12:21:20.816918 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:20.816920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:20.816983 (+ 63us) server_negotiation.cc:407] Connection header received
0928 12:21:20.817069 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:20.817070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:20.817081 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:20.817168 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:20.817568 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.818118 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.818889 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.819142 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.819208 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:20.820073 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:20.820074 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:20.820074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:20.820081 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:20.821193 (+ 1112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:20.821500 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:20.821501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:20.821502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:20.821554 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:20.821705 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:20.821706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:20.821706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:20.821857 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:20.821917 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:20.822075 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 12:21:20.822142 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 12:21:20 dist-test-slave-920p krb5kdc[3958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 12:21:20 dist-test-slave-920p krb5kdc[3958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:20.842566 4052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:20.836129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48614 (local address 127.2.56.62:39635)
0928 12:21:20.836179 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:20.836181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:20.836189 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:21:20.836220 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:20.836221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:20.836230 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:20.836280 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:20.836805 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.837382 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.838069 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.838254 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.838298 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:20.840410 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:20.840412 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:20.840412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:20.840419 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:20.841706 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:20.842001 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:20.842001 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:20.842002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:20.842046 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:20.842169 (+ 123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:20.842169 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:20.842169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:20.842267 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:20.842336 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:20.842419 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 12:21:20.842479 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 12:21:20.844198 4003 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:21:20.846349 4003 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:21:20.853848 4506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:20.849123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:46837 (local address 127.0.0.1:33692)
0928 12:21:20.849592 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:20.849601 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:20.849650 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:20.850012 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:20.850013 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:20.850014 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:20.850155 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:20.850159 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.851514 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.851514 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:20.852639 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:20.852641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.853567 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.853568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:20.853633 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:20.853637 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:21:20.853682 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 12:21:20.853726 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":41,"thread_start_us":107,"threads_started":1}
I20260928 12:21:20.853896 4505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:20.849234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48542 (local address 127.2.56.3:34425)
0928 12:21:20.849802 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:20.849807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:20.849817 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:21:20.849858 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:20.849860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:20.849877 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:20.849961 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:20.850236 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.851125 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.852749 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.853344 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.853419 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:20.853495 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:20.853608 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 12:21:20.853765 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":110,"threads_started":1}
I20260928 12:21:20.853778 4504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:20.849113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:34425 (local address 127.0.0.1:48542)
0928 12:21:20.849296 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:20.849343 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:20.849420 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:20.849991 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:20.849996 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:20.849998 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:20.850120 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:20.850123 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.851210 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.851210 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:20.852638 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:20.852641 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.853418 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.853419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:20.853500 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:20.853515 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 12:21:20.853585 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 12:21:20.853652 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":23,"thread_start_us":79,"threads_started":1}
I20260928 12:21:20.853992 4507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:20.849458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33692 (local address 127.2.56.2:46837)
0928 12:21:20.849774 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:20.849780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:20.849791 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:21:20.849836 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:20.849838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:20.849857 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:20.849942 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:20.850248 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.851421 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.852721 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.853434 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.853487 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:20.853533 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:20.853822 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 12:21:20.853867 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":106,"threads_started":1}
I20260928 12:21:20.854347 4508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:20.849596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47094 (local address 127.2.56.1:36185)
0928 12:21:20.850088 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:20.850092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:20.850199 (+ 107us) server_negotiation.cc:407] Connection header received
0928 12:21:20.850280 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:20.850282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:20.850307 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:20.850391 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:20.850517 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.851611 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.852624 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.853385 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.853467 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:20.853531 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:20.854206 (+ 675us) server_negotiation.cc:299] Negotiation successful
0928 12:21:20.854252 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":98,"threads_started":1}
I20260928 12:21:20.854547 4503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:20.849458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:36185 (local address 127.0.0.1:47094)
0928 12:21:20.850103 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:20.850112 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:20.850205 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:20.850400 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:20.850401 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:20.850402 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:20.850452 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:20.850453 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.851736 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.851737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:20.852536 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:20.852539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:20.853566 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:20.853567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:20.853829 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:20.853833 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:21:20.853876 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 12:21:20.854486 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":3,"thread_start_us":80,"threads_started":1}
I20260928 12:21:20.854786 4420 tablet_service.cc:1467] Processing CreateTablet for tablet 17ab04555ece46e8bc03322a74f3c2e9 (DEFAULT_TABLE table=test-table [id=f1a06965763640d489ad7635ca538325]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:20.855240 4420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17ab04555ece46e8bc03322a74f3c2e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:20.855177 4280 tablet_service.cc:1467] Processing CreateTablet for tablet 17ab04555ece46e8bc03322a74f3c2e9 (DEFAULT_TABLE table=test-table [id=f1a06965763640d489ad7635ca538325]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:20.855500 4280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17ab04555ece46e8bc03322a74f3c2e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:20.857965 4140 tablet_service.cc:1467] Processing CreateTablet for tablet 17ab04555ece46e8bc03322a74f3c2e9 (DEFAULT_TABLE table=test-table [id=f1a06965763640d489ad7635ca538325]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:20.858273 4140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17ab04555ece46e8bc03322a74f3c2e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:20.858564 4509 tablet_bootstrap.cc:492] T 17ab04555ece46e8bc03322a74f3c2e9 P a02cd150a0444b2a931e91e89cfa49a2: Bootstrap starting.
I20260928 12:21:20.859277 4509 tablet_bootstrap.cc:654] T 17ab04555ece46e8bc03322a74f3c2e9 P a02cd150a0444b2a931e91e89cfa49a2: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:20.859658 4509 log.cc:824] T 17ab04555ece46e8bc03322a74f3c2e9 P a02cd150a0444b2a931e91e89cfa49a2: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:20.860455 4509 tablet_bootstrap.cc:492] T 17ab04555ece46e8bc03322a74f3c2e9 P a02cd150a0444b2a931e91e89cfa49a2: No bootstrap required, opened a new log
I20260928 12:21:20.860525 4509 ts_tablet_manager.cc:1397] T 17ab04555ece46e8bc03322a74f3c2e9 P a02cd150a0444b2a931e91e89cfa49a2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:21:20.861912 4511 tablet_bootstrap.cc:492] T 17ab04555ece46e8bc03322a74f3c2e9 P 646a0acf9a8841009d4a3fe5b5e43150: Bootstrap starting.
I20260928 12:21:20.862123 4512 tablet_bootstrap.cc:492] T 17ab04555ece46e8bc03322a74f3c2e9 P dd4cf280ae1748ea8b67e9f08551e3bb: Bootstrap starting.
I20260928 12:21:20.862695 4509 raft_consensus.cc:348] T 17ab04555ece46e8bc03322a74f3c2e9 P a02cd150a0444b2a931e91e89cfa49a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } } peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } }
I20260928 12:21:20.862843 4509 raft_consensus.cc:374] T 17ab04555ece46e8bc03322a74f3c2e9 P a02cd150a0444b2a931e91e89cfa49a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:20.862874 4509 raft_consensus.cc:729] T 17ab04555ece46e8bc03322a74f3c2e9 P a02cd150a0444b2a931e91e89cfa49a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a02cd150a0444b2a931e91e89cfa49a2, State: Initialized, Role: FOLLOWER
I20260928 12:21:20.862874 4512 tablet_bootstrap.cc:654] T 17ab04555ece46e8bc03322a74f3c2e9 P dd4cf280ae1748ea8b67e9f08551e3bb: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:20.862874 4511 tablet_bootstrap.cc:654] T 17ab04555ece46e8bc03322a74f3c2e9 P 646a0acf9a8841009d4a3fe5b5e43150: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:20.863018 4509 consensus_queue.cc:260] T 17ab04555ece46e8bc03322a74f3c2e9 P a02cd150a0444b2a931e91e89cfa49a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } } peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } }
I20260928 12:21:20.863253 4511 log.cc:824] T 17ab04555ece46e8bc03322a74f3c2e9 P 646a0acf9a8841009d4a3fe5b5e43150: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:20.863253 4512 log.cc:824] T 17ab04555ece46e8bc03322a74f3c2e9 P dd4cf280ae1748ea8b67e9f08551e3bb: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:20.863307 4346 heartbeater.cc:503] Master 127.2.56.62:39635 was elected leader, sending a full tablet report...
I20260928 12:21:20.863302 4509 ts_tablet_manager.cc:1428] T 17ab04555ece46e8bc03322a74f3c2e9 P a02cd150a0444b2a931e91e89cfa49a2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:21:20.864269 4511 tablet_bootstrap.cc:492] T 17ab04555ece46e8bc03322a74f3c2e9 P 646a0acf9a8841009d4a3fe5b5e43150: No bootstrap required, opened a new log
I20260928 12:21:20.864269 4512 tablet_bootstrap.cc:492] T 17ab04555ece46e8bc03322a74f3c2e9 P dd4cf280ae1748ea8b67e9f08551e3bb: No bootstrap required, opened a new log
I20260928 12:21:20.864338 4512 ts_tablet_manager.cc:1397] T 17ab04555ece46e8bc03322a74f3c2e9 P dd4cf280ae1748ea8b67e9f08551e3bb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:21:20.864434 4511 ts_tablet_manager.cc:1397] T 17ab04555ece46e8bc03322a74f3c2e9 P 646a0acf9a8841009d4a3fe5b5e43150: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 12:21:20.866516 4511 raft_consensus.cc:348] T 17ab04555ece46e8bc03322a74f3c2e9 P 646a0acf9a8841009d4a3fe5b5e43150 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } } peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } }
I20260928 12:21:20.866516 4512 raft_consensus.cc:348] T 17ab04555ece46e8bc03322a74f3c2e9 P dd4cf280ae1748ea8b67e9f08551e3bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } } peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } }
I20260928 12:21:20.866726 4511 raft_consensus.cc:374] T 17ab04555ece46e8bc03322a74f3c2e9 P 646a0acf9a8841009d4a3fe5b5e43150 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:20.866726 4512 raft_consensus.cc:374] T 17ab04555ece46e8bc03322a74f3c2e9 P dd4cf280ae1748ea8b67e9f08551e3bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:20.866762 4512 raft_consensus.cc:729] T 17ab04555ece46e8bc03322a74f3c2e9 P dd4cf280ae1748ea8b67e9f08551e3bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd4cf280ae1748ea8b67e9f08551e3bb, State: Initialized, Role: FOLLOWER
I20260928 12:21:20.866762 4511 raft_consensus.cc:729] T 17ab04555ece46e8bc03322a74f3c2e9 P 646a0acf9a8841009d4a3fe5b5e43150 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 646a0acf9a8841009d4a3fe5b5e43150, State: Initialized, Role: FOLLOWER
I20260928 12:21:20.866901 4511 consensus_queue.cc:260] T 17ab04555ece46e8bc03322a74f3c2e9 P 646a0acf9a8841009d4a3fe5b5e43150 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } } peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } }
I20260928 12:21:20.866901 4512 consensus_queue.cc:260] T 17ab04555ece46e8bc03322a74f3c2e9 P dd4cf280ae1748ea8b67e9f08551e3bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } } peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } }
I20260928 12:21:20.867233 4511 ts_tablet_manager.cc:1428] T 17ab04555ece46e8bc03322a74f3c2e9 P 646a0acf9a8841009d4a3fe5b5e43150: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:21:20.867233 4512 ts_tablet_manager.cc:1428] T 17ab04555ece46e8bc03322a74f3c2e9 P dd4cf280ae1748ea8b67e9f08551e3bb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:21:20.867255 4206 heartbeater.cc:503] Master 127.2.56.62:39635 was elected leader, sending a full tablet report...
I20260928 12:21:20.867394 4486 heartbeater.cc:503] Master 127.2.56.62:39635 was elected leader, sending a full tablet report...
W20260928 12:21:20.896129 4347 tablet.cc:2369] T 17ab04555ece46e8bc03322a74f3c2e9 P a02cd150a0444b2a931e91e89cfa49a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:21:20.976519 4207 tablet.cc:2369] T 17ab04555ece46e8bc03322a74f3c2e9 P dd4cf280ae1748ea8b67e9f08551e3bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:21:21.063328 4487 tablet.cc:2369] T 17ab04555ece46e8bc03322a74f3c2e9 P 646a0acf9a8841009d4a3fe5b5e43150: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:21:21.127000 4513 raft_consensus.cc:482] T 17ab04555ece46e8bc03322a74f3c2e9 P a02cd150a0444b2a931e91e89cfa49a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:21:21.127174 4513 raft_consensus.cc:504] T 17ab04555ece46e8bc03322a74f3c2e9 P a02cd150a0444b2a931e91e89cfa49a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } } peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } }
I20260928 12:21:21.127581 4513 leader_election.cc:288] T 17ab04555ece46e8bc03322a74f3c2e9 P a02cd150a0444b2a931e91e89cfa49a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 646a0acf9a8841009d4a3fe5b5e43150 (127.2.56.3:34425), dd4cf280ae1748ea8b67e9f08551e3bb (127.2.56.1:36185)
I20260928 12:21:21.131824 4508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:21.128054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:56031 (local address 127.2.56.1:36185)
0928 12:21:21.128117 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:21.128119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:21.128148 (+ 29us) server_negotiation.cc:407] Connection header received
0928 12:21:21.128226 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:21.128227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:21.128238 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:21.128282 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:21.128428 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:21.129162 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:21.130836 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:21.131422 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:21.131494 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:21.131508 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:21.131698 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 12:21:21.131745 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 12:21:21.131824 4505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:21.127935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:59865 (local address 127.2.56.3:34425)
0928 12:21:21.127992 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:21.127995 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:21.128008 (+ 13us) server_negotiation.cc:407] Connection header received
0928 12:21:21.128048 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:21.128049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:21.128062 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:21.128149 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:21.128332 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:21.129137 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:21.130821 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:21.131426 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:21.131495 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:21.131508 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:21.131689 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 12:21:21.131736 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 12:21:21.131884 4239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:21.127835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:34425 (local address 127.2.56.2:59865)
0928 12:21:21.127908 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:21.127922 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:21.127980 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:21.128164 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:21.128165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:21.128167 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:21.128258 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:21.128260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:21.129247 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:21.129249 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:21.130707 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:21.130712 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:21.131517 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:21.131518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:21.131619 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:21.131622 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:21:21.131678 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 12:21:21.131737 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":139}
I20260928 12:21:21.131894 4518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:21.127959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:36185 (local address 127.2.56.2:56031)
0928 12:21:21.128094 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:21.128102 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:21.128164 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:21.128298 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:21.128299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:21.128300 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:21.128374 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:21.128375 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:21.129248 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:21.129249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:21.130719 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:21.130721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:21.131517 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:21.131518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:21.131611 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:21.131617 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 12:21:21.131678 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 12:21:21.131736 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":34,"thread_start_us":90,"threads_started":1}
I20260928 12:21:21.132326 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ab04555ece46e8bc03322a74f3c2e9" candidate_uuid: "a02cd150a0444b2a931e91e89cfa49a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "646a0acf9a8841009d4a3fe5b5e43150" is_pre_election: true
I20260928 12:21:21.132346 4160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ab04555ece46e8bc03322a74f3c2e9" candidate_uuid: "a02cd150a0444b2a931e91e89cfa49a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" is_pre_election: true
I20260928 12:21:21.132524 4160 raft_consensus.cc:2454] T 17ab04555ece46e8bc03322a74f3c2e9 P dd4cf280ae1748ea8b67e9f08551e3bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a02cd150a0444b2a931e91e89cfa49a2 in term 0.
I20260928 12:21:21.132525 4440 raft_consensus.cc:2454] T 17ab04555ece46e8bc03322a74f3c2e9 P 646a0acf9a8841009d4a3fe5b5e43150 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a02cd150a0444b2a931e91e89cfa49a2 in term 0.
I20260928 12:21:21.132756 4232 leader_election.cc:302] T 17ab04555ece46e8bc03322a74f3c2e9 P a02cd150a0444b2a931e91e89cfa49a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a02cd150a0444b2a931e91e89cfa49a2, dd4cf280ae1748ea8b67e9f08551e3bb; no voters:
I20260928 12:21:21.132895 4513 raft_consensus.cc:2781] T 17ab04555ece46e8bc03322a74f3c2e9 P a02cd150a0444b2a931e91e89cfa49a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:21:21.132956 4513 raft_consensus.cc:482] T 17ab04555ece46e8bc03322a74f3c2e9 P a02cd150a0444b2a931e91e89cfa49a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:21:21.132980 4513 raft_consensus.cc:3037] T 17ab04555ece46e8bc03322a74f3c2e9 P a02cd150a0444b2a931e91e89cfa49a2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:21.133857 4513 raft_consensus.cc:504] T 17ab04555ece46e8bc03322a74f3c2e9 P a02cd150a0444b2a931e91e89cfa49a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } } peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } }
I20260928 12:21:21.133996 4513 leader_election.cc:288] T 17ab04555ece46e8bc03322a74f3c2e9 P a02cd150a0444b2a931e91e89cfa49a2 [CANDIDATE]: Term 1 election: Requested vote from peers 646a0acf9a8841009d4a3fe5b5e43150 (127.2.56.3:34425), dd4cf280ae1748ea8b67e9f08551e3bb (127.2.56.1:36185)
I20260928 12:21:21.134153 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ab04555ece46e8bc03322a74f3c2e9" candidate_uuid: "a02cd150a0444b2a931e91e89cfa49a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "646a0acf9a8841009d4a3fe5b5e43150"
I20260928 12:21:21.134169 4160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ab04555ece46e8bc03322a74f3c2e9" candidate_uuid: "a02cd150a0444b2a931e91e89cfa49a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb"
I20260928 12:21:21.134228 4440 raft_consensus.cc:3037] T 17ab04555ece46e8bc03322a74f3c2e9 P 646a0acf9a8841009d4a3fe5b5e43150 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:21.134248 4160 raft_consensus.cc:3037] T 17ab04555ece46e8bc03322a74f3c2e9 P dd4cf280ae1748ea8b67e9f08551e3bb [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:21.135020 4160 raft_consensus.cc:2454] T 17ab04555ece46e8bc03322a74f3c2e9 P dd4cf280ae1748ea8b67e9f08551e3bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a02cd150a0444b2a931e91e89cfa49a2 in term 1.
I20260928 12:21:21.135083 4440 raft_consensus.cc:2454] T 17ab04555ece46e8bc03322a74f3c2e9 P 646a0acf9a8841009d4a3fe5b5e43150 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a02cd150a0444b2a931e91e89cfa49a2 in term 1.
I20260928 12:21:21.135258 4232 leader_election.cc:302] T 17ab04555ece46e8bc03322a74f3c2e9 P a02cd150a0444b2a931e91e89cfa49a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 646a0acf9a8841009d4a3fe5b5e43150, a02cd150a0444b2a931e91e89cfa49a2; no voters:
I20260928 12:21:21.135347 4513 raft_consensus.cc:2781] T 17ab04555ece46e8bc03322a74f3c2e9 P a02cd150a0444b2a931e91e89cfa49a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:21:21.135521 4513 raft_consensus.cc:686] T 17ab04555ece46e8bc03322a74f3c2e9 P a02cd150a0444b2a931e91e89cfa49a2 [term 1 LEADER]: Becoming Leader. State: Replica: a02cd150a0444b2a931e91e89cfa49a2, State: Running, Role: LEADER
I20260928 12:21:21.135637 4513 consensus_queue.cc:237] T 17ab04555ece46e8bc03322a74f3c2e9 P a02cd150a0444b2a931e91e89cfa49a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } } peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } }
I20260928 12:21:21.136579 4003 catalog_manager.cc:5495] T 17ab04555ece46e8bc03322a74f3c2e9 P a02cd150a0444b2a931e91e89cfa49a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a02cd150a0444b2a931e91e89cfa49a2 (127.2.56.2). New cstate: current_term: 1 leader_uuid: "a02cd150a0444b2a931e91e89cfa49a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } health_report { overall_health: HEALTHY } } }
Sep 28 12:21:21 dist-test-slave-920p krb5kdc[3958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598080, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:21.241287 4052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:21.234265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48630 (local address 127.2.56.62:39635)
0928 12:21:21.234330 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:21.234333 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:21.234494 (+ 161us) server_negotiation.cc:407] Connection header received
0928 12:21:21.234551 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:21.234552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:21.234563 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:21.234609 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:21.235166 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:21.235731 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:21.236472 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:21.236654 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:21.236701 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:21.238901 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:21.238902 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:21.238902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:21.238909 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:21.240201 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:21.240551 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:21.240551 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:21.240552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:21.240609 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:21.240766 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:21.240766 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:21.240767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:21.240875 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:21.240929 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:21.241071 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 12:21:21.241164 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 12:21:21.241379 4526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:21.234199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39635 (local address 127.0.0.1:48630)
0928 12:21:21.234412 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:21.234422 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:21.234504 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:21.234623 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:21.234623 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:21.235011 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:21.235089 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:21.235092 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:21.235832 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:21.235833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:21.236404 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:21.236406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:21.236703 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:21.236704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:21.236741 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:21.237317 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:21.237323 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:21.238826 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:21.240286 (+ 1460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:21.240293 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:21.240296 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:21.240488 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:21.240662 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:21.240662 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:21.240663 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:21.240719 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:21.240927 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:21.240935 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:21.240983 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 12:21:21.241043 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 12:21:21.241232 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":83,"thread_start_us":115,"threads_started":1}
I20260928 12:21:21.243346 4003 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:48630:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 12:21:21.243563 4003 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:21:21.245193 4280 tablet_service.cc:1467] Processing CreateTablet for tablet 5bed1e6ce6364abca061e2fffc60bbfe (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c0aba417c2ef436ea068a7ed3b5eb009]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 12:21:21.245213 4140 tablet_service.cc:1467] Processing CreateTablet for tablet 5bed1e6ce6364abca061e2fffc60bbfe (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c0aba417c2ef436ea068a7ed3b5eb009]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 12:21:21.245210 4420 tablet_service.cc:1467] Processing CreateTablet for tablet 5bed1e6ce6364abca061e2fffc60bbfe (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c0aba417c2ef436ea068a7ed3b5eb009]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 12:21:21.245425 4280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bed1e6ce6364abca061e2fffc60bbfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:21.245425 4420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bed1e6ce6364abca061e2fffc60bbfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:21.245425 4140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bed1e6ce6364abca061e2fffc60bbfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:21.246912 4512 tablet_bootstrap.cc:492] T 5bed1e6ce6364abca061e2fffc60bbfe P dd4cf280ae1748ea8b67e9f08551e3bb: Bootstrap starting.
I20260928 12:21:21.247188 4509 tablet_bootstrap.cc:492] T 5bed1e6ce6364abca061e2fffc60bbfe P a02cd150a0444b2a931e91e89cfa49a2: Bootstrap starting.
I20260928 12:21:21.247403 4512 tablet_bootstrap.cc:654] T 5bed1e6ce6364abca061e2fffc60bbfe P dd4cf280ae1748ea8b67e9f08551e3bb: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:21.247601 4509 tablet_bootstrap.cc:654] T 5bed1e6ce6364abca061e2fffc60bbfe P a02cd150a0444b2a931e91e89cfa49a2: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:21.248088 4512 tablet_bootstrap.cc:492] T 5bed1e6ce6364abca061e2fffc60bbfe P dd4cf280ae1748ea8b67e9f08551e3bb: No bootstrap required, opened a new log
I20260928 12:21:21.248142 4512 ts_tablet_manager.cc:1397] T 5bed1e6ce6364abca061e2fffc60bbfe P dd4cf280ae1748ea8b67e9f08551e3bb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:21:21.248250 4511 tablet_bootstrap.cc:492] T 5bed1e6ce6364abca061e2fffc60bbfe P 646a0acf9a8841009d4a3fe5b5e43150: Bootstrap starting.
I20260928 12:21:21.248327 4512 raft_consensus.cc:348] T 5bed1e6ce6364abca061e2fffc60bbfe P dd4cf280ae1748ea8b67e9f08551e3bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } }
I20260928 12:21:21.248413 4512 raft_consensus.cc:374] T 5bed1e6ce6364abca061e2fffc60bbfe P dd4cf280ae1748ea8b67e9f08551e3bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:21.248440 4512 raft_consensus.cc:729] T 5bed1e6ce6364abca061e2fffc60bbfe P dd4cf280ae1748ea8b67e9f08551e3bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd4cf280ae1748ea8b67e9f08551e3bb, State: Initialized, Role: FOLLOWER
I20260928 12:21:21.248509 4512 consensus_queue.cc:260] T 5bed1e6ce6364abca061e2fffc60bbfe P dd4cf280ae1748ea8b67e9f08551e3bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } }
I20260928 12:21:21.248631 4512 ts_tablet_manager.cc:1428] T 5bed1e6ce6364abca061e2fffc60bbfe P dd4cf280ae1748ea8b67e9f08551e3bb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:21.248706 4517 tablet_replica.cc:439] T 5bed1e6ce6364abca061e2fffc60bbfe P dd4cf280ae1748ea8b67e9f08551e3bb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } } }
I20260928 12:21:21.248778 4517 tablet_replica.cc:442] T 5bed1e6ce6364abca061e2fffc60bbfe P dd4cf280ae1748ea8b67e9f08551e3bb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:21:21.248842 4511 tablet_bootstrap.cc:654] T 5bed1e6ce6364abca061e2fffc60bbfe P 646a0acf9a8841009d4a3fe5b5e43150: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:21.249413 4509 tablet_bootstrap.cc:492] T 5bed1e6ce6364abca061e2fffc60bbfe P a02cd150a0444b2a931e91e89cfa49a2: No bootstrap required, opened a new log
I20260928 12:21:21.249471 4509 ts_tablet_manager.cc:1397] T 5bed1e6ce6364abca061e2fffc60bbfe P a02cd150a0444b2a931e91e89cfa49a2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:21:21.249637 4509 raft_consensus.cc:348] T 5bed1e6ce6364abca061e2fffc60bbfe P a02cd150a0444b2a931e91e89cfa49a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } }
I20260928 12:21:21.249703 4509 raft_consensus.cc:374] T 5bed1e6ce6364abca061e2fffc60bbfe P a02cd150a0444b2a931e91e89cfa49a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:21.249716 4509 raft_consensus.cc:729] T 5bed1e6ce6364abca061e2fffc60bbfe P a02cd150a0444b2a931e91e89cfa49a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a02cd150a0444b2a931e91e89cfa49a2, State: Initialized, Role: FOLLOWER
I20260928 12:21:21.249756 4509 consensus_queue.cc:260] T 5bed1e6ce6364abca061e2fffc60bbfe P a02cd150a0444b2a931e91e89cfa49a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } }
I20260928 12:21:21.249845 4509 ts_tablet_manager.cc:1428] T 5bed1e6ce6364abca061e2fffc60bbfe P a02cd150a0444b2a931e91e89cfa49a2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:21.249918 4513 tablet_replica.cc:439] T 5bed1e6ce6364abca061e2fffc60bbfe P a02cd150a0444b2a931e91e89cfa49a2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } } }
I20260928 12:21:21.249993 4513 tablet_replica.cc:442] T 5bed1e6ce6364abca061e2fffc60bbfe P a02cd150a0444b2a931e91e89cfa49a2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:21:21.250270 4511 tablet_bootstrap.cc:492] T 5bed1e6ce6364abca061e2fffc60bbfe P 646a0acf9a8841009d4a3fe5b5e43150: No bootstrap required, opened a new log
I20260928 12:21:21.250324 4511 ts_tablet_manager.cc:1397] T 5bed1e6ce6364abca061e2fffc60bbfe P 646a0acf9a8841009d4a3fe5b5e43150: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:21:21.250496 4511 raft_consensus.cc:348] T 5bed1e6ce6364abca061e2fffc60bbfe P 646a0acf9a8841009d4a3fe5b5e43150 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } }
I20260928 12:21:21.250576 4511 raft_consensus.cc:374] T 5bed1e6ce6364abca061e2fffc60bbfe P 646a0acf9a8841009d4a3fe5b5e43150 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:21.250597 4511 raft_consensus.cc:729] T 5bed1e6ce6364abca061e2fffc60bbfe P 646a0acf9a8841009d4a3fe5b5e43150 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 646a0acf9a8841009d4a3fe5b5e43150, State: Initialized, Role: FOLLOWER
I20260928 12:21:21.250646 4511 consensus_queue.cc:260] T 5bed1e6ce6364abca061e2fffc60bbfe P 646a0acf9a8841009d4a3fe5b5e43150 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } }
I20260928 12:21:21.250739 4511 ts_tablet_manager.cc:1428] T 5bed1e6ce6364abca061e2fffc60bbfe P 646a0acf9a8841009d4a3fe5b5e43150: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:21.250824 4516 tablet_replica.cc:439] T 5bed1e6ce6364abca061e2fffc60bbfe P 646a0acf9a8841009d4a3fe5b5e43150: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } } }
I20260928 12:21:21.250912 4516 tablet_replica.cc:442] T 5bed1e6ce6364abca061e2fffc60bbfe P 646a0acf9a8841009d4a3fe5b5e43150: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:21:21.277180 4517 raft_consensus.cc:482] T 5bed1e6ce6364abca061e2fffc60bbfe P dd4cf280ae1748ea8b67e9f08551e3bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:21:21.277307 4517 raft_consensus.cc:504] T 5bed1e6ce6364abca061e2fffc60bbfe P dd4cf280ae1748ea8b67e9f08551e3bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } }
I20260928 12:21:21.277707 4517 leader_election.cc:288] T 5bed1e6ce6364abca061e2fffc60bbfe P dd4cf280ae1748ea8b67e9f08551e3bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a02cd150a0444b2a931e91e89cfa49a2 (127.2.56.2:46837), 646a0acf9a8841009d4a3fe5b5e43150 (127.2.56.3:34425)
I20260928 12:21:21.281121 4530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:21.277914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:34425 (local address 127.2.56.1:47435)
0928 12:21:21.278091 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:21.278101 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:21.278160 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:21.278348 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:21.278349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:21.278352 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:21.278450 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:21.278453 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:21.279112 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:21.279113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:21.280110 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:21.280113 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:21.280891 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:21.280892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:21.280946 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:21.280951 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:21:21.280991 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 12:21:21.281027 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":45,"spinlock_wait_cycles":768,"thread_start_us":99,"threads_started":1}
I20260928 12:21:21.281124 4505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:21.278011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:47435 (local address 127.2.56.3:34425)
0928 12:21:21.278058 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:21.278060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:21.278163 (+ 103us) server_negotiation.cc:407] Connection header received
0928 12:21:21.278227 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:21.278228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:21.278240 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:21.278296 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:21.278515 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:21.279044 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:21.280211 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:21.280812 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:21.280886 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:21.280900 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:21.281011 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 12:21:21.281050 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 12:21:21.281389 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bed1e6ce6364abca061e2fffc60bbfe" candidate_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "646a0acf9a8841009d4a3fe5b5e43150" is_pre_election: true
I20260928 12:21:21.281503 4440 raft_consensus.cc:2454] T 5bed1e6ce6364abca061e2fffc60bbfe P 646a0acf9a8841009d4a3fe5b5e43150 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd4cf280ae1748ea8b67e9f08551e3bb in term 0.
I20260928 12:21:21.281706 4092 leader_election.cc:302] T 5bed1e6ce6364abca061e2fffc60bbfe P dd4cf280ae1748ea8b67e9f08551e3bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 646a0acf9a8841009d4a3fe5b5e43150, dd4cf280ae1748ea8b67e9f08551e3bb; no voters:
I20260928 12:21:21.281890 4517 raft_consensus.cc:2781] T 5bed1e6ce6364abca061e2fffc60bbfe P dd4cf280ae1748ea8b67e9f08551e3bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:21:21.281929 4517 raft_consensus.cc:482] T 5bed1e6ce6364abca061e2fffc60bbfe P dd4cf280ae1748ea8b67e9f08551e3bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:21:21.281947 4517 raft_consensus.cc:3037] T 5bed1e6ce6364abca061e2fffc60bbfe P dd4cf280ae1748ea8b67e9f08551e3bb [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:21.282048 4531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:21.277914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:46837 (local address 127.2.56.1:49685)
0928 12:21:21.278167 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:21.278178 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:21.278248 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:21.278405 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:21.278406 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:21.278407 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:21.278481 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:21.278483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:21.279182 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:21.279183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:21.281065 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:21.281068 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:21.281797 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:21.281798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:21.281868 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:21.281874 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 12:21:21.281923 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 12:21:21.281962 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":27,"thread_start_us":96,"threads_started":1}
I20260928 12:21:21.282035 4507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:21.277964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:49685 (local address 127.2.56.2:46837)
0928 12:21:21.278019 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:21.278021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:21.278262 (+ 241us) server_negotiation.cc:407] Connection header received
0928 12:21:21.278329 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:21.278330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:21.278343 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:21.278396 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:21.278547 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:21.279104 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:21.281157 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:21.281719 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:21.281779 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:21.281790 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:21.281928 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 12:21:21.281966 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 12:21:21.282390 4300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bed1e6ce6364abca061e2fffc60bbfe" candidate_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a02cd150a0444b2a931e91e89cfa49a2" is_pre_election: true
I20260928 12:21:21.282505 4300 raft_consensus.cc:2454] T 5bed1e6ce6364abca061e2fffc60bbfe P a02cd150a0444b2a931e91e89cfa49a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd4cf280ae1748ea8b67e9f08551e3bb in term 0.
I20260928 12:21:21.282923 4517 raft_consensus.cc:504] T 5bed1e6ce6364abca061e2fffc60bbfe P dd4cf280ae1748ea8b67e9f08551e3bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } }
I20260928 12:21:21.283061 4517 leader_election.cc:288] T 5bed1e6ce6364abca061e2fffc60bbfe P dd4cf280ae1748ea8b67e9f08551e3bb [CANDIDATE]: Term 1 election: Requested vote from peers a02cd150a0444b2a931e91e89cfa49a2 (127.2.56.2:46837), 646a0acf9a8841009d4a3fe5b5e43150 (127.2.56.3:34425)
I20260928 12:21:21.283219 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bed1e6ce6364abca061e2fffc60bbfe" candidate_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "646a0acf9a8841009d4a3fe5b5e43150"
I20260928 12:21:21.283241 4300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bed1e6ce6364abca061e2fffc60bbfe" candidate_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a02cd150a0444b2a931e91e89cfa49a2"
I20260928 12:21:21.283286 4440 raft_consensus.cc:3037] T 5bed1e6ce6364abca061e2fffc60bbfe P 646a0acf9a8841009d4a3fe5b5e43150 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:21.283304 4300 raft_consensus.cc:3037] T 5bed1e6ce6364abca061e2fffc60bbfe P a02cd150a0444b2a931e91e89cfa49a2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:21.283938 4440 raft_consensus.cc:2454] T 5bed1e6ce6364abca061e2fffc60bbfe P 646a0acf9a8841009d4a3fe5b5e43150 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd4cf280ae1748ea8b67e9f08551e3bb in term 1.
I20260928 12:21:21.284026 4300 raft_consensus.cc:2454] T 5bed1e6ce6364abca061e2fffc60bbfe P a02cd150a0444b2a931e91e89cfa49a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd4cf280ae1748ea8b67e9f08551e3bb in term 1.
I20260928 12:21:21.284089 4092 leader_election.cc:302] T 5bed1e6ce6364abca061e2fffc60bbfe P dd4cf280ae1748ea8b67e9f08551e3bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 646a0acf9a8841009d4a3fe5b5e43150, dd4cf280ae1748ea8b67e9f08551e3bb; no voters:
I20260928 12:21:21.284173 4517 raft_consensus.cc:2781] T 5bed1e6ce6364abca061e2fffc60bbfe P dd4cf280ae1748ea8b67e9f08551e3bb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:21:21.284291 4517 raft_consensus.cc:686] T 5bed1e6ce6364abca061e2fffc60bbfe P dd4cf280ae1748ea8b67e9f08551e3bb [term 1 LEADER]: Becoming Leader. State: Replica: dd4cf280ae1748ea8b67e9f08551e3bb, State: Running, Role: LEADER
I20260928 12:21:21.284384 4517 consensus_queue.cc:237] T 5bed1e6ce6364abca061e2fffc60bbfe P dd4cf280ae1748ea8b67e9f08551e3bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } }
I20260928 12:21:21.284876 4532 tablet_replica.cc:439] T 5bed1e6ce6364abca061e2fffc60bbfe P dd4cf280ae1748ea8b67e9f08551e3bb: TxnStatusTablet state changed. Reason: New leader dd4cf280ae1748ea8b67e9f08551e3bb. Latest consensus state: current_term: 1 leader_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } } }
I20260928 12:21:21.284986 4532 tablet_replica.cc:442] T 5bed1e6ce6364abca061e2fffc60bbfe P dd4cf280ae1748ea8b67e9f08551e3bb: This TxnStatusTablet replica's current role is: LEADER
I20260928 12:21:21.285228 4534 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 12:21:21.285382 4534 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 12:21:21.285295 4003 catalog_manager.cc:5495] T 5bed1e6ce6364abca061e2fffc60bbfe P dd4cf280ae1748ea8b67e9f08551e3bb reported cstate change: term changed from 0 to 1, leader changed from <none> to dd4cf280ae1748ea8b67e9f08551e3bb (127.2.56.1). New cstate: current_term: 1 leader_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } health_report { overall_health: UNKNOWN } } }
I20260928 12:21:21.296895 4508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:21.293900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47102 (local address 127.2.56.1:36185)
0928 12:21:21.293945 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:21.293947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:21.294002 (+ 55us) server_negotiation.cc:407] Connection header received
0928 12:21:21.294105 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:21.294106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:21.294124 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:21.294178 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:21:21.294309 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:21.294900 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:21.296092 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:21.296318 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:21.296365 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:21.296553 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:21:21.296660 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:21:21.296713 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:21.296767 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 12:21:21.296788 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:21:21.296922 4526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:21.293860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:36185 (local address 127.0.0.1:47102)
0928 12:21:21.293957 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:21.293969 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:21.294025 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:21.294190 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:21.294191 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:21.294193 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:21:21.294261 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:21.294264 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:21.294979 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:21.294981 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:21.296013 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:21.296017 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:21.296369 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:21.296370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:21.296444 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:21.296466 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:21:21.296707 (+ 241us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:21:21.296712 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:21:21.296764 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 12:21:21.296831 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 12:21:21.299083 4440 raft_consensus.cc:1261] T 5bed1e6ce6364abca061e2fffc60bbfe P 646a0acf9a8841009d4a3fe5b5e43150 [term 1 FOLLOWER]: Refusing update from remote peer dd4cf280ae1748ea8b67e9f08551e3bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:21:21.299088 4300 raft_consensus.cc:1261] T 5bed1e6ce6364abca061e2fffc60bbfe P a02cd150a0444b2a931e91e89cfa49a2 [term 1 FOLLOWER]: Refusing update from remote peer dd4cf280ae1748ea8b67e9f08551e3bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:21:21.299417 4517 consensus_queue.cc:1035] T 5bed1e6ce6364abca061e2fffc60bbfe P dd4cf280ae1748ea8b67e9f08551e3bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:21:21.299635 4517 consensus_queue.cc:1035] T 5bed1e6ce6364abca061e2fffc60bbfe P dd4cf280ae1748ea8b67e9f08551e3bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:21:21.300881 4513 tablet_replica.cc:439] T 5bed1e6ce6364abca061e2fffc60bbfe P a02cd150a0444b2a931e91e89cfa49a2: TxnStatusTablet state changed. Reason: New leader dd4cf280ae1748ea8b67e9f08551e3bb. Latest consensus state: current_term: 1 leader_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } } }
I20260928 12:21:21.300907 4516 tablet_replica.cc:439] T 5bed1e6ce6364abca061e2fffc60bbfe P 646a0acf9a8841009d4a3fe5b5e43150: TxnStatusTablet state changed. Reason: New leader dd4cf280ae1748ea8b67e9f08551e3bb. Latest consensus state: current_term: 1 leader_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } } }
I20260928 12:21:21.300989 4513 tablet_replica.cc:442] T 5bed1e6ce6364abca061e2fffc60bbfe P a02cd150a0444b2a931e91e89cfa49a2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:21:21.300992 4516 tablet_replica.cc:442] T 5bed1e6ce6364abca061e2fffc60bbfe P 646a0acf9a8841009d4a3fe5b5e43150: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:21:21.302188 4517 tablet_replica.cc:439] T 5bed1e6ce6364abca061e2fffc60bbfe P dd4cf280ae1748ea8b67e9f08551e3bb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } } }
I20260928 12:21:21.302277 4517 tablet_replica.cc:442] T 5bed1e6ce6364abca061e2fffc60bbfe P dd4cf280ae1748ea8b67e9f08551e3bb: This TxnStatusTablet replica's current role is: LEADER
I20260928 12:21:21.302251 4513 tablet_replica.cc:439] T 5bed1e6ce6364abca061e2fffc60bbfe P a02cd150a0444b2a931e91e89cfa49a2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } } }
I20260928 12:21:21.302331 4513 tablet_replica.cc:442] T 5bed1e6ce6364abca061e2fffc60bbfe P a02cd150a0444b2a931e91e89cfa49a2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:21:21.302358 4539 mvcc.cc:205] Tried to move back new op lower bound from 7334289740998246400 to 7334289740941434880. Current Snapshot: MvccSnapshot[applied={T|T < 7334289740998246400}]
I20260928 12:21:21.302521 4537 mvcc.cc:205] Tried to move back new op lower bound from 7334289740998246400 to 7334289740941434880. Current Snapshot: MvccSnapshot[applied={T|T < 7334289740998246400}]
I20260928 12:21:21.302799 4532 tablet_replica.cc:439] T 5bed1e6ce6364abca061e2fffc60bbfe P dd4cf280ae1748ea8b67e9f08551e3bb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } } }
I20260928 12:21:21.302871 4532 tablet_replica.cc:442] T 5bed1e6ce6364abca061e2fffc60bbfe P dd4cf280ae1748ea8b67e9f08551e3bb: This TxnStatusTablet replica's current role is: LEADER
I20260928 12:21:21.303454 4516 tablet_replica.cc:439] T 5bed1e6ce6364abca061e2fffc60bbfe P 646a0acf9a8841009d4a3fe5b5e43150: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } } }
I20260928 12:21:21.303547 4516 tablet_replica.cc:442] T 5bed1e6ce6364abca061e2fffc60bbfe P 646a0acf9a8841009d4a3fe5b5e43150: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:21:21.303773 4538 mvcc.cc:205] Tried to move back new op lower bound from 7334289740998246400 to 7334289740941434880. Current Snapshot: MvccSnapshot[applied={T|T < 7334289740998246400}]
I20260928 12:21:21.308439 4507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:21.305766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33704 (local address 127.2.56.2:46837)
0928 12:21:21.305811 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:21.305813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:21.305845 (+ 32us) server_negotiation.cc:407] Connection header received
0928 12:21:21.305931 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:21.305931 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:21.305958 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:21.306026 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:21:21.306191 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:21.306812 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:21.307617 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:21.307882 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:21.307938 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:21.308101 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:21:21.308225 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:21:21.308295 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:21.308332 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 12:21:21.308360 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 12:21:21.313609 4508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:21.311151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:40555 (local address 127.2.56.1:36185)
0928 12:21:21.311195 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:21.311197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:21.311322 (+ 125us) server_negotiation.cc:407] Connection header received
0928 12:21:21.311377 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:21.311378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:21.311390 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:21.311436 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:21:21.311587 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:21.312088 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:21.312881 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:21.313103 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:21.313159 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:21.313371 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:21:21.313404 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:21:21.313444 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:21.313500 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 12:21:21.313523 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:21:21.313634 4548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:21.311088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:36185 (local address 127.2.56.2:40555)
0928 12:21:21.311256 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:21.311267 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:21.311327 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:21.311437 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:21.311438 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:21.311440 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:21:21.311534 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:21.311536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:21.312159 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:21.312160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:21.312807 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:21.312809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:21.313166 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:21.313167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:21.313232 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:21.313247 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:21:21.313448 (+ 201us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:21:21.313458 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 12:21:21.313497 (+ 39us) client_negotiation.cc:240] Negotiation successful
0928 12:21:21.313537 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":96,"threads_started":1}
I20260928 12:21:21.316638 4440 raft_consensus.cc:1261] T 17ab04555ece46e8bc03322a74f3c2e9 P 646a0acf9a8841009d4a3fe5b5e43150 [term 1 FOLLOWER]: Refusing update from remote peer a02cd150a0444b2a931e91e89cfa49a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:21:21.316716 4160 raft_consensus.cc:1261] T 17ab04555ece46e8bc03322a74f3c2e9 P dd4cf280ae1748ea8b67e9f08551e3bb [term 1 FOLLOWER]: Refusing update from remote peer a02cd150a0444b2a931e91e89cfa49a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:21:21.316890 4513 consensus_queue.cc:1035] T 17ab04555ece46e8bc03322a74f3c2e9 P a02cd150a0444b2a931e91e89cfa49a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:21:21.317070 4519 consensus_queue.cc:1035] T 17ab04555ece46e8bc03322a74f3c2e9 P a02cd150a0444b2a931e91e89cfa49a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:21:21.326340 4553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:21.323339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:46837 (local address 127.2.56.1:32939)
0928 12:21:21.323518 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:21.323532 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:21.323590 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:21.323711 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:21.323712 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:21.323713 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:21:21.323780 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:21.323781 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:21.324644 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:21.324645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:21.325492 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:21.325495 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:21.325970 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:21.325971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:21.326018 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:21.326033 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:21:21.326187 (+ 154us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:21:21.326189 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:21:21.326219 (+ 30us) client_negotiation.cc:240] Negotiation successful
0928 12:21:21.326254 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":102,"threads_started":1}
I20260928 12:21:21.326333 4507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:21.323555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:32939 (local address 127.2.56.2:46837)
0928 12:21:21.323599 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:21.323601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:21.323611 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:21:21.323636 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:21.323637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:21.323649 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:21.323710 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:21:21.323824 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:21.324570 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:21.325578 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:21.325897 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:21.325963 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:21.326101 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:21:21.326139 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:21:21.326188 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:21.326228 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 12:21:21.326248 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:21:21.335217 4002 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:21:21.335363 4002 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:21:21.336226 4002 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4796d476b91e4f5998165aec0ca20110: Sending DeleteTablet for 3 replicas of tablet 17ab04555ece46e8bc03322a74f3c2e9
I20260928 12:21:21.336747 4420 tablet_service.cc:1514] Processing DeleteTablet for tablet 17ab04555ece46e8bc03322a74f3c2e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:21 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:48542
I20260928 12:21:21.336957 4140 tablet_service.cc:1514] Processing DeleteTablet for tablet 17ab04555ece46e8bc03322a74f3c2e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:21 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:47094
I20260928 12:21:21.337108 4558 tablet_replica.cc:330] T 17ab04555ece46e8bc03322a74f3c2e9 P 646a0acf9a8841009d4a3fe5b5e43150: stopping tablet replica
I20260928 12:21:21.337172 4280 tablet_service.cc:1514] Processing DeleteTablet for tablet 17ab04555ece46e8bc03322a74f3c2e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:21 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:33692
I20260928 12:21:21.337360 4558 raft_consensus.cc:2229] T 17ab04555ece46e8bc03322a74f3c2e9 P 646a0acf9a8841009d4a3fe5b5e43150 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:21:21.337476 4558 raft_consensus.cc:2258] T 17ab04555ece46e8bc03322a74f3c2e9 P 646a0acf9a8841009d4a3fe5b5e43150 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:21.337795 4002 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:21:21.337980 4002 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:21:21.338258 4558 ts_tablet_manager.cc:1905] T 17ab04555ece46e8bc03322a74f3c2e9 P 646a0acf9a8841009d4a3fe5b5e43150: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:21.339578 4140 tablet_service.cc:1467] Processing CreateTablet for tablet 3af3901460a3482eaac887178d601803 (DEFAULT_TABLE table=test-table [id=e1ac6673589547ef921db2e927777bbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:21.339722 4140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3af3901460a3482eaac887178d601803. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:21.340488 4558 ts_tablet_manager.cc:1918] T 17ab04555ece46e8bc03322a74f3c2e9 P 646a0acf9a8841009d4a3fe5b5e43150: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:21:21.340633 4558 log.cc:1196] T 17ab04555ece46e8bc03322a74f3c2e9 P 646a0acf9a8841009d4a3fe5b5e43150: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-2/wal/wals/17ab04555ece46e8bc03322a74f3c2e9
I20260928 12:21:21.340987 4558 ts_tablet_manager.cc:1939] T 17ab04555ece46e8bc03322a74f3c2e9 P 646a0acf9a8841009d4a3fe5b5e43150: Deleting consensus metadata
I20260928 12:21:21.341254 4512 tablet_bootstrap.cc:492] T 3af3901460a3482eaac887178d601803 P dd4cf280ae1748ea8b67e9f08551e3bb: Bootstrap starting.
I20260928 12:21:21.341588 3988 catalog_manager.cc:4841] TS 646a0acf9a8841009d4a3fe5b5e43150 (127.2.56.3:34425): tablet 17ab04555ece46e8bc03322a74f3c2e9 (table test-table [id=f1a06965763640d489ad7635ca538325]) successfully deleted
I20260928 12:21:21.341717 4512 tablet_bootstrap.cc:654] T 3af3901460a3482eaac887178d601803 P dd4cf280ae1748ea8b67e9f08551e3bb: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:21.341837 4559 tablet_replica.cc:330] T 17ab04555ece46e8bc03322a74f3c2e9 P dd4cf280ae1748ea8b67e9f08551e3bb: stopping tablet replica
I20260928 12:21:21.342041 4559 raft_consensus.cc:2229] T 17ab04555ece46e8bc03322a74f3c2e9 P dd4cf280ae1748ea8b67e9f08551e3bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:21:21.342175 4559 raft_consensus.cc:2258] T 17ab04555ece46e8bc03322a74f3c2e9 P dd4cf280ae1748ea8b67e9f08551e3bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:21.342227 4420 tablet_service.cc:1467] Processing CreateTablet for tablet 3af3901460a3482eaac887178d601803 (DEFAULT_TABLE table=test-table [id=e1ac6673589547ef921db2e927777bbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:21.342340 4420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3af3901460a3482eaac887178d601803. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:21.342800 4559 ts_tablet_manager.cc:1905] T 17ab04555ece46e8bc03322a74f3c2e9 P dd4cf280ae1748ea8b67e9f08551e3bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:21.343567 4511 tablet_bootstrap.cc:492] T 3af3901460a3482eaac887178d601803 P 646a0acf9a8841009d4a3fe5b5e43150: Bootstrap starting.
I20260928 12:21:21.343621 4280 tablet_service.cc:1467] Processing CreateTablet for tablet 3af3901460a3482eaac887178d601803 (DEFAULT_TABLE table=test-table [id=e1ac6673589547ef921db2e927777bbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:21.343739 4280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3af3901460a3482eaac887178d601803. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:21.343982 4511 tablet_bootstrap.cc:654] T 3af3901460a3482eaac887178d601803 P 646a0acf9a8841009d4a3fe5b5e43150: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:21.345206 4509 tablet_bootstrap.cc:492] T 3af3901460a3482eaac887178d601803 P a02cd150a0444b2a931e91e89cfa49a2: Bootstrap starting.
I20260928 12:21:21.345630 4559 ts_tablet_manager.cc:1918] T 17ab04555ece46e8bc03322a74f3c2e9 P dd4cf280ae1748ea8b67e9f08551e3bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:21:21.345669 4509 tablet_bootstrap.cc:654] T 3af3901460a3482eaac887178d601803 P a02cd150a0444b2a931e91e89cfa49a2: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:21.345762 4559 log.cc:1196] T 17ab04555ece46e8bc03322a74f3c2e9 P dd4cf280ae1748ea8b67e9f08551e3bb: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-0/wal/wals/17ab04555ece46e8bc03322a74f3c2e9
I20260928 12:21:21.346101 4559 ts_tablet_manager.cc:1939] T 17ab04555ece46e8bc03322a74f3c2e9 P dd4cf280ae1748ea8b67e9f08551e3bb: Deleting consensus metadata
I20260928 12:21:21.346572 3988 catalog_manager.cc:4841] TS dd4cf280ae1748ea8b67e9f08551e3bb (127.2.56.1:36185): tablet 17ab04555ece46e8bc03322a74f3c2e9 (table test-table [id=f1a06965763640d489ad7635ca538325]) successfully deleted
I20260928 12:21:21.346920 4512 tablet_bootstrap.cc:492] T 3af3901460a3482eaac887178d601803 P dd4cf280ae1748ea8b67e9f08551e3bb: No bootstrap required, opened a new log
I20260928 12:21:21.346971 4512 ts_tablet_manager.cc:1397] T 3af3901460a3482eaac887178d601803 P dd4cf280ae1748ea8b67e9f08551e3bb: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260928 12:21:21.347014 4511 tablet_bootstrap.cc:492] T 3af3901460a3482eaac887178d601803 P 646a0acf9a8841009d4a3fe5b5e43150: No bootstrap required, opened a new log
I20260928 12:21:21.347059 4511 ts_tablet_manager.cc:1397] T 3af3901460a3482eaac887178d601803 P 646a0acf9a8841009d4a3fe5b5e43150: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 12:21:21.347128 4512 raft_consensus.cc:348] T 3af3901460a3482eaac887178d601803 P dd4cf280ae1748ea8b67e9f08551e3bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } }
I20260928 12:21:21.347203 4512 raft_consensus.cc:374] T 3af3901460a3482eaac887178d601803 P dd4cf280ae1748ea8b67e9f08551e3bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:21.347222 4512 raft_consensus.cc:729] T 3af3901460a3482eaac887178d601803 P dd4cf280ae1748ea8b67e9f08551e3bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd4cf280ae1748ea8b67e9f08551e3bb, State: Initialized, Role: FOLLOWER
I20260928 12:21:21.347206 4511 raft_consensus.cc:348] T 3af3901460a3482eaac887178d601803 P 646a0acf9a8841009d4a3fe5b5e43150 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } }
I20260928 12:21:21.347273 4511 raft_consensus.cc:374] T 3af3901460a3482eaac887178d601803 P 646a0acf9a8841009d4a3fe5b5e43150 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:21.347268 4512 consensus_queue.cc:260] T 3af3901460a3482eaac887178d601803 P dd4cf280ae1748ea8b67e9f08551e3bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } }
I20260928 12:21:21.347292 4511 raft_consensus.cc:729] T 3af3901460a3482eaac887178d601803 P 646a0acf9a8841009d4a3fe5b5e43150 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 646a0acf9a8841009d4a3fe5b5e43150, State: Initialized, Role: FOLLOWER
I20260928 12:21:21.347335 4511 consensus_queue.cc:260] T 3af3901460a3482eaac887178d601803 P 646a0acf9a8841009d4a3fe5b5e43150 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } }
I20260928 12:21:21.347354 4512 ts_tablet_manager.cc:1428] T 3af3901460a3482eaac887178d601803 P dd4cf280ae1748ea8b67e9f08551e3bb: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 12:21:21.347416 4511 ts_tablet_manager.cc:1428] T 3af3901460a3482eaac887178d601803 P 646a0acf9a8841009d4a3fe5b5e43150: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:21.349151 4509 tablet_bootstrap.cc:492] T 3af3901460a3482eaac887178d601803 P a02cd150a0444b2a931e91e89cfa49a2: No bootstrap required, opened a new log
I20260928 12:21:21.349228 4509 ts_tablet_manager.cc:1397] T 3af3901460a3482eaac887178d601803 P a02cd150a0444b2a931e91e89cfa49a2: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260928 12:21:21.349412 4509 raft_consensus.cc:348] T 3af3901460a3482eaac887178d601803 P a02cd150a0444b2a931e91e89cfa49a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } }
I20260928 12:21:21.349514 4509 raft_consensus.cc:374] T 3af3901460a3482eaac887178d601803 P a02cd150a0444b2a931e91e89cfa49a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:21.349534 4509 raft_consensus.cc:729] T 3af3901460a3482eaac887178d601803 P a02cd150a0444b2a931e91e89cfa49a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a02cd150a0444b2a931e91e89cfa49a2, State: Initialized, Role: FOLLOWER
I20260928 12:21:21.349577 4509 consensus_queue.cc:260] T 3af3901460a3482eaac887178d601803 P a02cd150a0444b2a931e91e89cfa49a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } }
I20260928 12:21:21.349671 4509 ts_tablet_manager.cc:1428] T 3af3901460a3482eaac887178d601803 P a02cd150a0444b2a931e91e89cfa49a2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:21.349864 4560 tablet_replica.cc:330] T 17ab04555ece46e8bc03322a74f3c2e9 P a02cd150a0444b2a931e91e89cfa49a2: stopping tablet replica
I20260928 12:21:21.349972 4560 raft_consensus.cc:2229] T 17ab04555ece46e8bc03322a74f3c2e9 P a02cd150a0444b2a931e91e89cfa49a2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:21:21.350569 4560 raft_consensus.cc:2258] T 17ab04555ece46e8bc03322a74f3c2e9 P a02cd150a0444b2a931e91e89cfa49a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:21.351097 4560 ts_tablet_manager.cc:1905] T 17ab04555ece46e8bc03322a74f3c2e9 P a02cd150a0444b2a931e91e89cfa49a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:21.353317 4560 ts_tablet_manager.cc:1918] T 17ab04555ece46e8bc03322a74f3c2e9 P a02cd150a0444b2a931e91e89cfa49a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:21:21.353394 4560 log.cc:1196] T 17ab04555ece46e8bc03322a74f3c2e9 P a02cd150a0444b2a931e91e89cfa49a2: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-1/wal/wals/17ab04555ece46e8bc03322a74f3c2e9
I20260928 12:21:21.353633 4560 ts_tablet_manager.cc:1939] T 17ab04555ece46e8bc03322a74f3c2e9 P a02cd150a0444b2a931e91e89cfa49a2: Deleting consensus metadata
I20260928 12:21:21.354053 3990 catalog_manager.cc:4841] TS a02cd150a0444b2a931e91e89cfa49a2 (127.2.56.2:46837): tablet 17ab04555ece46e8bc03322a74f3c2e9 (table test-table [id=f1a06965763640d489ad7635ca538325]) successfully deleted
I20260928 12:21:21.402920 4554 raft_consensus.cc:482] T 3af3901460a3482eaac887178d601803 P a02cd150a0444b2a931e91e89cfa49a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:21:21.403066 4554 raft_consensus.cc:504] T 3af3901460a3482eaac887178d601803 P a02cd150a0444b2a931e91e89cfa49a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } }
I20260928 12:21:21.403290 4554 leader_election.cc:288] T 3af3901460a3482eaac887178d601803 P a02cd150a0444b2a931e91e89cfa49a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd4cf280ae1748ea8b67e9f08551e3bb (127.2.56.1:36185), 646a0acf9a8841009d4a3fe5b5e43150 (127.2.56.3:34425)
I20260928 12:21:21.403582 4160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af3901460a3482eaac887178d601803" candidate_uuid: "a02cd150a0444b2a931e91e89cfa49a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" is_pre_election: true
I20260928 12:21:21.403578 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af3901460a3482eaac887178d601803" candidate_uuid: "a02cd150a0444b2a931e91e89cfa49a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "646a0acf9a8841009d4a3fe5b5e43150" is_pre_election: true
I20260928 12:21:21.403692 4160 raft_consensus.cc:2454] T 3af3901460a3482eaac887178d601803 P dd4cf280ae1748ea8b67e9f08551e3bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a02cd150a0444b2a931e91e89cfa49a2 in term 0.
I20260928 12:21:21.403704 4440 raft_consensus.cc:2454] T 3af3901460a3482eaac887178d601803 P 646a0acf9a8841009d4a3fe5b5e43150 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a02cd150a0444b2a931e91e89cfa49a2 in term 0.
I20260928 12:21:21.403885 4232 leader_election.cc:302] T 3af3901460a3482eaac887178d601803 P a02cd150a0444b2a931e91e89cfa49a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a02cd150a0444b2a931e91e89cfa49a2, dd4cf280ae1748ea8b67e9f08551e3bb; no voters:
I20260928 12:21:21.404073 4554 raft_consensus.cc:2781] T 3af3901460a3482eaac887178d601803 P a02cd150a0444b2a931e91e89cfa49a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:21:21.404122 4554 raft_consensus.cc:482] T 3af3901460a3482eaac887178d601803 P a02cd150a0444b2a931e91e89cfa49a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:21:21.404141 4554 raft_consensus.cc:3037] T 3af3901460a3482eaac887178d601803 P a02cd150a0444b2a931e91e89cfa49a2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:21.405234 4554 raft_consensus.cc:504] T 3af3901460a3482eaac887178d601803 P a02cd150a0444b2a931e91e89cfa49a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } }
I20260928 12:21:21.405404 4554 leader_election.cc:288] T 3af3901460a3482eaac887178d601803 P a02cd150a0444b2a931e91e89cfa49a2 [CANDIDATE]: Term 1 election: Requested vote from peers dd4cf280ae1748ea8b67e9f08551e3bb (127.2.56.1:36185), 646a0acf9a8841009d4a3fe5b5e43150 (127.2.56.3:34425)
I20260928 12:21:21.405622 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af3901460a3482eaac887178d601803" candidate_uuid: "a02cd150a0444b2a931e91e89cfa49a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "646a0acf9a8841009d4a3fe5b5e43150"
I20260928 12:21:21.405709 4440 raft_consensus.cc:3037] T 3af3901460a3482eaac887178d601803 P 646a0acf9a8841009d4a3fe5b5e43150 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:21.405876 4160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af3901460a3482eaac887178d601803" candidate_uuid: "a02cd150a0444b2a931e91e89cfa49a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb"
I20260928 12:21:21.405978 4160 raft_consensus.cc:3037] T 3af3901460a3482eaac887178d601803 P dd4cf280ae1748ea8b67e9f08551e3bb [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:21.406641 4440 raft_consensus.cc:2454] T 3af3901460a3482eaac887178d601803 P 646a0acf9a8841009d4a3fe5b5e43150 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a02cd150a0444b2a931e91e89cfa49a2 in term 1.
I20260928 12:21:21.406816 4232 leader_election.cc:302] T 3af3901460a3482eaac887178d601803 P a02cd150a0444b2a931e91e89cfa49a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 646a0acf9a8841009d4a3fe5b5e43150, a02cd150a0444b2a931e91e89cfa49a2; no voters:
I20260928 12:21:21.406893 4554 raft_consensus.cc:2781] T 3af3901460a3482eaac887178d601803 P a02cd150a0444b2a931e91e89cfa49a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:21:21.406925 4554 raft_consensus.cc:686] T 3af3901460a3482eaac887178d601803 P a02cd150a0444b2a931e91e89cfa49a2 [term 1 LEADER]: Becoming Leader. State: Replica: a02cd150a0444b2a931e91e89cfa49a2, State: Running, Role: LEADER
I20260928 12:21:21.406997 4160 raft_consensus.cc:2454] T 3af3901460a3482eaac887178d601803 P dd4cf280ae1748ea8b67e9f08551e3bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a02cd150a0444b2a931e91e89cfa49a2 in term 1.
I20260928 12:21:21.406984 4554 consensus_queue.cc:237] T 3af3901460a3482eaac887178d601803 P a02cd150a0444b2a931e91e89cfa49a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } }
I20260928 12:21:21.407704 4002 catalog_manager.cc:5495] T 3af3901460a3482eaac887178d601803 P a02cd150a0444b2a931e91e89cfa49a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a02cd150a0444b2a931e91e89cfa49a2 (127.2.56.2). New cstate: current_term: 1 leader_uuid: "a02cd150a0444b2a931e91e89cfa49a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } health_report { overall_health: UNKNOWN } } }
I20260928 12:21:21.427127 4160 raft_consensus.cc:1261] T 3af3901460a3482eaac887178d601803 P dd4cf280ae1748ea8b67e9f08551e3bb [term 1 FOLLOWER]: Refusing update from remote peer a02cd150a0444b2a931e91e89cfa49a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:21:21.427625 4440 raft_consensus.cc:1261] T 3af3901460a3482eaac887178d601803 P 646a0acf9a8841009d4a3fe5b5e43150 [term 1 FOLLOWER]: Refusing update from remote peer a02cd150a0444b2a931e91e89cfa49a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:21:21.427736 4519 consensus_queue.cc:1035] T 3af3901460a3482eaac887178d601803 P a02cd150a0444b2a931e91e89cfa49a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:21:21.428048 4519 consensus_queue.cc:1035] T 3af3901460a3482eaac887178d601803 P a02cd150a0444b2a931e91e89cfa49a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 12:21:21.434432 4539 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 12:21:22.955295 4002 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:21:22.955451 4002 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:21:22.956408 4002 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4796d476b91e4f5998165aec0ca20110: Sending DeleteTablet for 3 replicas of tablet 3af3901460a3482eaac887178d601803
I20260928 12:21:22.956676 4140 tablet_service.cc:1514] Processing DeleteTablet for tablet 3af3901460a3482eaac887178d601803 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:22 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:47094
I20260928 12:21:22.956698 4280 tablet_service.cc:1514] Processing DeleteTablet for tablet 3af3901460a3482eaac887178d601803 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:22 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:33692
I20260928 12:21:22.956913 4420 tablet_service.cc:1514] Processing DeleteTablet for tablet 3af3901460a3482eaac887178d601803 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:22 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:48542
I20260928 12:21:22.957044 4589 tablet_replica.cc:330] T 3af3901460a3482eaac887178d601803 P a02cd150a0444b2a931e91e89cfa49a2: stopping tablet replica
I20260928 12:21:22.957118 4589 raft_consensus.cc:2229] T 3af3901460a3482eaac887178d601803 P a02cd150a0444b2a931e91e89cfa49a2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:21:22.957166 4590 tablet_replica.cc:330] T 3af3901460a3482eaac887178d601803 P 646a0acf9a8841009d4a3fe5b5e43150: stopping tablet replica
I20260928 12:21:22.957192 4589 raft_consensus.cc:2258] T 3af3901460a3482eaac887178d601803 P a02cd150a0444b2a931e91e89cfa49a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:22.957227 4590 raft_consensus.cc:2229] T 3af3901460a3482eaac887178d601803 P 646a0acf9a8841009d4a3fe5b5e43150 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:21:22.957294 4590 raft_consensus.cc:2258] T 3af3901460a3482eaac887178d601803 P 646a0acf9a8841009d4a3fe5b5e43150 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:22.957516 4589 ts_tablet_manager.cc:1905] T 3af3901460a3482eaac887178d601803 P a02cd150a0444b2a931e91e89cfa49a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:22.959174 4589 ts_tablet_manager.cc:1918] T 3af3901460a3482eaac887178d601803 P a02cd150a0444b2a931e91e89cfa49a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 12:21:22.959231 4589 log.cc:1196] T 3af3901460a3482eaac887178d601803 P a02cd150a0444b2a931e91e89cfa49a2: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-1/wal/wals/3af3901460a3482eaac887178d601803
I20260928 12:21:22.959477 4589 ts_tablet_manager.cc:1939] T 3af3901460a3482eaac887178d601803 P a02cd150a0444b2a931e91e89cfa49a2: Deleting consensus metadata
I20260928 12:21:22.959815 3990 catalog_manager.cc:4841] TS a02cd150a0444b2a931e91e89cfa49a2 (127.2.56.2:46837): tablet 3af3901460a3482eaac887178d601803 (table test-table [id=e1ac6673589547ef921db2e927777bbe]) successfully deleted
I20260928 12:21:22.960314 4590 ts_tablet_manager.cc:1905] T 3af3901460a3482eaac887178d601803 P 646a0acf9a8841009d4a3fe5b5e43150: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:22.960657 4588 tablet_replica.cc:330] T 3af3901460a3482eaac887178d601803 P dd4cf280ae1748ea8b67e9f08551e3bb: stopping tablet replica
I20260928 12:21:22.960733 4588 raft_consensus.cc:2229] T 3af3901460a3482eaac887178d601803 P dd4cf280ae1748ea8b67e9f08551e3bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:21:22.960783 4588 raft_consensus.cc:2258] T 3af3901460a3482eaac887178d601803 P dd4cf280ae1748ea8b67e9f08551e3bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:22.961048 4588 ts_tablet_manager.cc:1905] T 3af3901460a3482eaac887178d601803 P dd4cf280ae1748ea8b67e9f08551e3bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:22.962113 4590 ts_tablet_manager.cc:1918] T 3af3901460a3482eaac887178d601803 P 646a0acf9a8841009d4a3fe5b5e43150: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 12:21:22.962182 4590 log.cc:1196] T 3af3901460a3482eaac887178d601803 P 646a0acf9a8841009d4a3fe5b5e43150: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-2/wal/wals/3af3901460a3482eaac887178d601803
I20260928 12:21:22.962389 4590 ts_tablet_manager.cc:1939] T 3af3901460a3482eaac887178d601803 P 646a0acf9a8841009d4a3fe5b5e43150: Deleting consensus metadata
I20260928 12:21:22.962881 4588 ts_tablet_manager.cc:1918] T 3af3901460a3482eaac887178d601803 P dd4cf280ae1748ea8b67e9f08551e3bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 12:21:22.962942 4588 log.cc:1196] T 3af3901460a3482eaac887178d601803 P dd4cf280ae1748ea8b67e9f08551e3bb: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-0/wal/wals/3af3901460a3482eaac887178d601803
I20260928 12:21:22.962978 3988 catalog_manager.cc:4841] TS 646a0acf9a8841009d4a3fe5b5e43150 (127.2.56.3:34425): tablet 3af3901460a3482eaac887178d601803 (table test-table [id=e1ac6673589547ef921db2e927777bbe]) successfully deleted
I20260928 12:21:22.963138 4588 ts_tablet_manager.cc:1939] T 3af3901460a3482eaac887178d601803 P dd4cf280ae1748ea8b67e9f08551e3bb: Deleting consensus metadata
I20260928 12:21:22.963762 3988 catalog_manager.cc:4841] TS dd4cf280ae1748ea8b67e9f08551e3bb (127.2.56.1:36185): tablet 3af3901460a3482eaac887178d601803 (table test-table [id=e1ac6673589547ef921db2e927777bbe]) successfully deleted
Sep 28 12:21:22 dist-test-slave-920p krb5kdc[3958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598082, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 12:21:22 dist-test-slave-920p krb5kdc[3958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598082, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:22.983146 4598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:22.975543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48644 (local address 127.2.56.62:39635)
0928 12:21:22.975734 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:22.975736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:22.975777 (+ 41us) server_negotiation.cc:407] Connection header received
0928 12:21:22.975821 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:22.975822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:22.975835 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:22.975892 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:22.976502 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:22.977088 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:22.977936 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:22.978143 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:22.978195 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:22.980681 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:22.980683 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:22.980683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:22.980691 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:22.982059 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:22.982389 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:22.982390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:22.982391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:22.982438 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:22.982654 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:22.982654 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:22.982655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:22.982767 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:22.982845 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:22.982981 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 12:21:22.983050 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":119,"threads_started":1}
I20260928 12:21:22.984800 4002 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:21:22.984983 4002 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:21:22.986537 4280 tablet_service.cc:1467] Processing CreateTablet for tablet e34be456744c4088872e220524e5ece1 (DEFAULT_TABLE table=test-table [id=1b209cf78cb74639bdc7596dc53e525c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:22.986529 4140 tablet_service.cc:1467] Processing CreateTablet for tablet e34be456744c4088872e220524e5ece1 (DEFAULT_TABLE table=test-table [id=1b209cf78cb74639bdc7596dc53e525c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:22.986594 4420 tablet_service.cc:1467] Processing CreateTablet for tablet e34be456744c4088872e220524e5ece1 (DEFAULT_TABLE table=test-table [id=1b209cf78cb74639bdc7596dc53e525c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:22.986694 4140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e34be456744c4088872e220524e5ece1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:22.986694 4280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e34be456744c4088872e220524e5ece1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:22.986706 4420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e34be456744c4088872e220524e5ece1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:22.987921 4599 tablet_bootstrap.cc:492] T e34be456744c4088872e220524e5ece1 P a02cd150a0444b2a931e91e89cfa49a2: Bootstrap starting.
I20260928 12:21:22.988255 4601 tablet_bootstrap.cc:492] T e34be456744c4088872e220524e5ece1 P 646a0acf9a8841009d4a3fe5b5e43150: Bootstrap starting.
I20260928 12:21:22.988407 4600 tablet_bootstrap.cc:492] T e34be456744c4088872e220524e5ece1 P dd4cf280ae1748ea8b67e9f08551e3bb: Bootstrap starting.
I20260928 12:21:22.988538 4599 tablet_bootstrap.cc:654] T e34be456744c4088872e220524e5ece1 P a02cd150a0444b2a931e91e89cfa49a2: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:22.988647 4601 tablet_bootstrap.cc:654] T e34be456744c4088872e220524e5ece1 P 646a0acf9a8841009d4a3fe5b5e43150: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:22.988833 4600 tablet_bootstrap.cc:654] T e34be456744c4088872e220524e5ece1 P dd4cf280ae1748ea8b67e9f08551e3bb: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:22.989212 4601 tablet_bootstrap.cc:492] T e34be456744c4088872e220524e5ece1 P 646a0acf9a8841009d4a3fe5b5e43150: No bootstrap required, opened a new log
I20260928 12:21:22.989289 4601 ts_tablet_manager.cc:1397] T e34be456744c4088872e220524e5ece1 P 646a0acf9a8841009d4a3fe5b5e43150: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:21:22.989475 4600 tablet_bootstrap.cc:492] T e34be456744c4088872e220524e5ece1 P dd4cf280ae1748ea8b67e9f08551e3bb: No bootstrap required, opened a new log
I20260928 12:21:22.989475 4601 raft_consensus.cc:348] T e34be456744c4088872e220524e5ece1 P 646a0acf9a8841009d4a3fe5b5e43150 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } }
I20260928 12:21:22.989521 4600 ts_tablet_manager.cc:1397] T e34be456744c4088872e220524e5ece1 P dd4cf280ae1748ea8b67e9f08551e3bb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:21:22.989548 4601 raft_consensus.cc:374] T e34be456744c4088872e220524e5ece1 P 646a0acf9a8841009d4a3fe5b5e43150 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:22.989573 4601 raft_consensus.cc:729] T e34be456744c4088872e220524e5ece1 P 646a0acf9a8841009d4a3fe5b5e43150 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 646a0acf9a8841009d4a3fe5b5e43150, State: Initialized, Role: FOLLOWER
I20260928 12:21:22.989622 4601 consensus_queue.cc:260] T e34be456744c4088872e220524e5ece1 P 646a0acf9a8841009d4a3fe5b5e43150 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } }
I20260928 12:21:22.989667 4600 raft_consensus.cc:348] T e34be456744c4088872e220524e5ece1 P dd4cf280ae1748ea8b67e9f08551e3bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } }
I20260928 12:21:22.989741 4600 raft_consensus.cc:374] T e34be456744c4088872e220524e5ece1 P dd4cf280ae1748ea8b67e9f08551e3bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:22.989764 4600 raft_consensus.cc:729] T e34be456744c4088872e220524e5ece1 P dd4cf280ae1748ea8b67e9f08551e3bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd4cf280ae1748ea8b67e9f08551e3bb, State: Initialized, Role: FOLLOWER
I20260928 12:21:22.989792 4601 ts_tablet_manager.cc:1428] T e34be456744c4088872e220524e5ece1 P 646a0acf9a8841009d4a3fe5b5e43150: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:22.989826 4600 consensus_queue.cc:260] T e34be456744c4088872e220524e5ece1 P dd4cf280ae1748ea8b67e9f08551e3bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } }
I20260928 12:21:22.990044 4599 tablet_bootstrap.cc:492] T e34be456744c4088872e220524e5ece1 P a02cd150a0444b2a931e91e89cfa49a2: No bootstrap required, opened a new log
I20260928 12:21:22.990094 4599 ts_tablet_manager.cc:1397] T e34be456744c4088872e220524e5ece1 P a02cd150a0444b2a931e91e89cfa49a2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:21:22.990226 4599 raft_consensus.cc:348] T e34be456744c4088872e220524e5ece1 P a02cd150a0444b2a931e91e89cfa49a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } }
I20260928 12:21:22.990305 4599 raft_consensus.cc:374] T e34be456744c4088872e220524e5ece1 P a02cd150a0444b2a931e91e89cfa49a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:22.990324 4599 raft_consensus.cc:729] T e34be456744c4088872e220524e5ece1 P a02cd150a0444b2a931e91e89cfa49a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a02cd150a0444b2a931e91e89cfa49a2, State: Initialized, Role: FOLLOWER
I20260928 12:21:22.990384 4599 consensus_queue.cc:260] T e34be456744c4088872e220524e5ece1 P a02cd150a0444b2a931e91e89cfa49a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } }
I20260928 12:21:22.990432 4600 ts_tablet_manager.cc:1428] T e34be456744c4088872e220524e5ece1 P dd4cf280ae1748ea8b67e9f08551e3bb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:21:22.990538 4599 ts_tablet_manager.cc:1428] T e34be456744c4088872e220524e5ece1 P a02cd150a0444b2a931e91e89cfa49a2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:23.003413 4579 raft_consensus.cc:482] T e34be456744c4088872e220524e5ece1 P a02cd150a0444b2a931e91e89cfa49a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:21:23.003521 4579 raft_consensus.cc:504] T e34be456744c4088872e220524e5ece1 P a02cd150a0444b2a931e91e89cfa49a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } }
I20260928 12:21:23.003693 4579 leader_election.cc:288] T e34be456744c4088872e220524e5ece1 P a02cd150a0444b2a931e91e89cfa49a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd4cf280ae1748ea8b67e9f08551e3bb (127.2.56.1:36185), 646a0acf9a8841009d4a3fe5b5e43150 (127.2.56.3:34425)
I20260928 12:21:23.003904 4160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e34be456744c4088872e220524e5ece1" candidate_uuid: "a02cd150a0444b2a931e91e89cfa49a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" is_pre_election: true
I20260928 12:21:23.003909 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e34be456744c4088872e220524e5ece1" candidate_uuid: "a02cd150a0444b2a931e91e89cfa49a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "646a0acf9a8841009d4a3fe5b5e43150" is_pre_election: true
I20260928 12:21:23.004019 4440 raft_consensus.cc:2454] T e34be456744c4088872e220524e5ece1 P 646a0acf9a8841009d4a3fe5b5e43150 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a02cd150a0444b2a931e91e89cfa49a2 in term 0.
I20260928 12:21:23.004019 4160 raft_consensus.cc:2454] T e34be456744c4088872e220524e5ece1 P dd4cf280ae1748ea8b67e9f08551e3bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a02cd150a0444b2a931e91e89cfa49a2 in term 0.
I20260928 12:21:23.004171 4232 leader_election.cc:302] T e34be456744c4088872e220524e5ece1 P a02cd150a0444b2a931e91e89cfa49a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 646a0acf9a8841009d4a3fe5b5e43150, a02cd150a0444b2a931e91e89cfa49a2; no voters:
I20260928 12:21:23.004274 4579 raft_consensus.cc:2781] T e34be456744c4088872e220524e5ece1 P a02cd150a0444b2a931e91e89cfa49a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:21:23.004313 4579 raft_consensus.cc:482] T e34be456744c4088872e220524e5ece1 P a02cd150a0444b2a931e91e89cfa49a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:21:23.004338 4579 raft_consensus.cc:3037] T e34be456744c4088872e220524e5ece1 P a02cd150a0444b2a931e91e89cfa49a2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:23.005137 4579 raft_consensus.cc:504] T e34be456744c4088872e220524e5ece1 P a02cd150a0444b2a931e91e89cfa49a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } }
I20260928 12:21:23.005254 4579 leader_election.cc:288] T e34be456744c4088872e220524e5ece1 P a02cd150a0444b2a931e91e89cfa49a2 [CANDIDATE]: Term 1 election: Requested vote from peers dd4cf280ae1748ea8b67e9f08551e3bb (127.2.56.1:36185), 646a0acf9a8841009d4a3fe5b5e43150 (127.2.56.3:34425)
I20260928 12:21:23.005465 4160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e34be456744c4088872e220524e5ece1" candidate_uuid: "a02cd150a0444b2a931e91e89cfa49a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb"
I20260928 12:21:23.005491 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e34be456744c4088872e220524e5ece1" candidate_uuid: "a02cd150a0444b2a931e91e89cfa49a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "646a0acf9a8841009d4a3fe5b5e43150"
I20260928 12:21:23.005565 4440 raft_consensus.cc:3037] T e34be456744c4088872e220524e5ece1 P 646a0acf9a8841009d4a3fe5b5e43150 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:23.005569 4160 raft_consensus.cc:3037] T e34be456744c4088872e220524e5ece1 P dd4cf280ae1748ea8b67e9f08551e3bb [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:23.006203 4440 raft_consensus.cc:2454] T e34be456744c4088872e220524e5ece1 P 646a0acf9a8841009d4a3fe5b5e43150 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a02cd150a0444b2a931e91e89cfa49a2 in term 1.
I20260928 12:21:23.006233 4160 raft_consensus.cc:2454] T e34be456744c4088872e220524e5ece1 P dd4cf280ae1748ea8b67e9f08551e3bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a02cd150a0444b2a931e91e89cfa49a2 in term 1.
I20260928 12:21:23.006377 4232 leader_election.cc:302] T e34be456744c4088872e220524e5ece1 P a02cd150a0444b2a931e91e89cfa49a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 646a0acf9a8841009d4a3fe5b5e43150, a02cd150a0444b2a931e91e89cfa49a2; no voters:
I20260928 12:21:23.006487 4579 raft_consensus.cc:2781] T e34be456744c4088872e220524e5ece1 P a02cd150a0444b2a931e91e89cfa49a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:21:23.006552 4579 raft_consensus.cc:686] T e34be456744c4088872e220524e5ece1 P a02cd150a0444b2a931e91e89cfa49a2 [term 1 LEADER]: Becoming Leader. State: Replica: a02cd150a0444b2a931e91e89cfa49a2, State: Running, Role: LEADER
I20260928 12:21:23.006614 4579 consensus_queue.cc:237] T e34be456744c4088872e220524e5ece1 P a02cd150a0444b2a931e91e89cfa49a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } }
I20260928 12:21:23.007318 4002 catalog_manager.cc:5495] T e34be456744c4088872e220524e5ece1 P a02cd150a0444b2a931e91e89cfa49a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a02cd150a0444b2a931e91e89cfa49a2 (127.2.56.2). New cstate: current_term: 1 leader_uuid: "a02cd150a0444b2a931e91e89cfa49a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } health_report { overall_health: UNKNOWN } } }
I20260928 12:21:23.015311 4608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:23.012846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33718 (local address 127.2.56.2:46837)
0928 12:21:23.013000 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:23.013002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:23.013012 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:21:23.013034 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:23.013034 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:23.013046 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:23.013121 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:21:23.013256 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:23.013868 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:23.014649 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:23.014892 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:23.014952 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:23.015096 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:21:23.015127 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:21:23.015167 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:23.015207 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 12:21:23.015229 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20260928 12:21:23.017661 4160 raft_consensus.cc:1261] T e34be456744c4088872e220524e5ece1 P dd4cf280ae1748ea8b67e9f08551e3bb [term 1 FOLLOWER]: Refusing update from remote peer a02cd150a0444b2a931e91e89cfa49a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:21:23.017678 4440 raft_consensus.cc:1261] T e34be456744c4088872e220524e5ece1 P 646a0acf9a8841009d4a3fe5b5e43150 [term 1 FOLLOWER]: Refusing update from remote peer a02cd150a0444b2a931e91e89cfa49a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:21:23.017865 4579 consensus_queue.cc:1035] T e34be456744c4088872e220524e5ece1 P a02cd150a0444b2a931e91e89cfa49a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:21:23.018028 4583 consensus_queue.cc:1035] T e34be456744c4088872e220524e5ece1 P a02cd150a0444b2a931e91e89cfa49a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:21:23.027688 4002 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:21:23.027812 4002 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:21:23.028399 4002 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4796d476b91e4f5998165aec0ca20110: Sending DeleteTablet for 3 replicas of tablet e34be456744c4088872e220524e5ece1
I20260928 12:21:23.028643 4140 tablet_service.cc:1514] Processing DeleteTablet for tablet e34be456744c4088872e220524e5ece1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:23 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:47094
I20260928 12:21:23.028644 4280 tablet_service.cc:1514] Processing DeleteTablet for tablet e34be456744c4088872e220524e5ece1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:23 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:33692
I20260928 12:21:23.028762 4420 tablet_service.cc:1514] Processing DeleteTablet for tablet e34be456744c4088872e220524e5ece1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:23 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:48542
I20260928 12:21:23.028762 4588 tablet_replica.cc:330] T e34be456744c4088872e220524e5ece1 P dd4cf280ae1748ea8b67e9f08551e3bb: stopping tablet replica
I20260928 12:21:23.028870 4588 raft_consensus.cc:2229] T e34be456744c4088872e220524e5ece1 P dd4cf280ae1748ea8b67e9f08551e3bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:21:23.028924 4588 raft_consensus.cc:2258] T e34be456744c4088872e220524e5ece1 P dd4cf280ae1748ea8b67e9f08551e3bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:23.028949 4590 tablet_replica.cc:330] T e34be456744c4088872e220524e5ece1 P 646a0acf9a8841009d4a3fe5b5e43150: stopping tablet replica
I20260928 12:21:23.029006 4589 tablet_replica.cc:330] T e34be456744c4088872e220524e5ece1 P a02cd150a0444b2a931e91e89cfa49a2: stopping tablet replica
I20260928 12:21:23.029014 4590 raft_consensus.cc:2229] T e34be456744c4088872e220524e5ece1 P 646a0acf9a8841009d4a3fe5b5e43150 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:21:23.029067 4589 raft_consensus.cc:2229] T e34be456744c4088872e220524e5ece1 P a02cd150a0444b2a931e91e89cfa49a2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:21:23.029075 4590 raft_consensus.cc:2258] T e34be456744c4088872e220524e5ece1 P 646a0acf9a8841009d4a3fe5b5e43150 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:23.029126 4589 raft_consensus.cc:2258] T e34be456744c4088872e220524e5ece1 P a02cd150a0444b2a931e91e89cfa49a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:23.029064 4002 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:21:23.029201 4002 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:21:23.029469 4588 ts_tablet_manager.cc:1905] T e34be456744c4088872e220524e5ece1 P dd4cf280ae1748ea8b67e9f08551e3bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:23.029592 4590 ts_tablet_manager.cc:1905] T e34be456744c4088872e220524e5ece1 P 646a0acf9a8841009d4a3fe5b5e43150: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:23.029601 4589 ts_tablet_manager.cc:1905] T e34be456744c4088872e220524e5ece1 P a02cd150a0444b2a931e91e89cfa49a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:23.030444 4420 tablet_service.cc:1467] Processing CreateTablet for tablet 1a40b4c8b29c458788f28214ddff3177 (DEFAULT_TABLE table=test-table [id=5572e1e7f8104707912f32c6854d73ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:23.030553 4420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a40b4c8b29c458788f28214ddff3177. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:23.031476 4589 ts_tablet_manager.cc:1918] T e34be456744c4088872e220524e5ece1 P a02cd150a0444b2a931e91e89cfa49a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:21:23.031493 4590 ts_tablet_manager.cc:1918] T e34be456744c4088872e220524e5ece1 P 646a0acf9a8841009d4a3fe5b5e43150: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:21:23.031533 4589 log.cc:1196] T e34be456744c4088872e220524e5ece1 P a02cd150a0444b2a931e91e89cfa49a2: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-1/wal/wals/e34be456744c4088872e220524e5ece1
I20260928 12:21:23.031538 4590 log.cc:1196] T e34be456744c4088872e220524e5ece1 P 646a0acf9a8841009d4a3fe5b5e43150: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-2/wal/wals/e34be456744c4088872e220524e5ece1
I20260928 12:21:23.031735 4589 ts_tablet_manager.cc:1939] T e34be456744c4088872e220524e5ece1 P a02cd150a0444b2a931e91e89cfa49a2: Deleting consensus metadata
I20260928 12:21:23.031745 4590 ts_tablet_manager.cc:1939] T e34be456744c4088872e220524e5ece1 P 646a0acf9a8841009d4a3fe5b5e43150: Deleting consensus metadata
I20260928 12:21:23.031764 4601 tablet_bootstrap.cc:492] T 1a40b4c8b29c458788f28214ddff3177 P 646a0acf9a8841009d4a3fe5b5e43150: Bootstrap starting.
I20260928 12:21:23.032027 4280 tablet_service.cc:1467] Processing CreateTablet for tablet 1a40b4c8b29c458788f28214ddff3177 (DEFAULT_TABLE table=test-table [id=5572e1e7f8104707912f32c6854d73ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:23.032047 4588 ts_tablet_manager.cc:1918] T e34be456744c4088872e220524e5ece1 P dd4cf280ae1748ea8b67e9f08551e3bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:21:23.032087 4588 log.cc:1196] T e34be456744c4088872e220524e5ece1 P dd4cf280ae1748ea8b67e9f08551e3bb: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-0/wal/wals/e34be456744c4088872e220524e5ece1
I20260928 12:21:23.032116 4280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a40b4c8b29c458788f28214ddff3177. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:23.032115 3990 catalog_manager.cc:4841] TS a02cd150a0444b2a931e91e89cfa49a2 (127.2.56.2:46837): tablet e34be456744c4088872e220524e5ece1 (table test-table [id=1b209cf78cb74639bdc7596dc53e525c]) successfully deleted
I20260928 12:21:23.032263 4588 ts_tablet_manager.cc:1939] T e34be456744c4088872e220524e5ece1 P dd4cf280ae1748ea8b67e9f08551e3bb: Deleting consensus metadata
I20260928 12:21:23.032550 3988 catalog_manager.cc:4841] TS dd4cf280ae1748ea8b67e9f08551e3bb (127.2.56.1:36185): tablet e34be456744c4088872e220524e5ece1 (table test-table [id=1b209cf78cb74639bdc7596dc53e525c]) successfully deleted
I20260928 12:21:23.032784 4601 tablet_bootstrap.cc:654] T 1a40b4c8b29c458788f28214ddff3177 P 646a0acf9a8841009d4a3fe5b5e43150: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:23.033059 4599 tablet_bootstrap.cc:492] T 1a40b4c8b29c458788f28214ddff3177 P a02cd150a0444b2a931e91e89cfa49a2: Bootstrap starting.
I20260928 12:21:23.033126 3988 catalog_manager.cc:4841] TS 646a0acf9a8841009d4a3fe5b5e43150 (127.2.56.3:34425): tablet e34be456744c4088872e220524e5ece1 (table test-table [id=1b209cf78cb74639bdc7596dc53e525c]) successfully deleted
I20260928 12:21:23.033301 4140 tablet_service.cc:1467] Processing CreateTablet for tablet 1a40b4c8b29c458788f28214ddff3177 (DEFAULT_TABLE table=test-table [id=5572e1e7f8104707912f32c6854d73ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:23.033383 4140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a40b4c8b29c458788f28214ddff3177. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:23.033396 4599 tablet_bootstrap.cc:654] T 1a40b4c8b29c458788f28214ddff3177 P a02cd150a0444b2a931e91e89cfa49a2: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:23.033957 4599 tablet_bootstrap.cc:492] T 1a40b4c8b29c458788f28214ddff3177 P a02cd150a0444b2a931e91e89cfa49a2: No bootstrap required, opened a new log
I20260928 12:21:23.034006 4599 ts_tablet_manager.cc:1397] T 1a40b4c8b29c458788f28214ddff3177 P a02cd150a0444b2a931e91e89cfa49a2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:21:23.034112 4601 tablet_bootstrap.cc:492] T 1a40b4c8b29c458788f28214ddff3177 P 646a0acf9a8841009d4a3fe5b5e43150: No bootstrap required, opened a new log
I20260928 12:21:23.034159 4601 ts_tablet_manager.cc:1397] T 1a40b4c8b29c458788f28214ddff3177 P 646a0acf9a8841009d4a3fe5b5e43150: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:21:23.034142 4599 raft_consensus.cc:348] T 1a40b4c8b29c458788f28214ddff3177 P a02cd150a0444b2a931e91e89cfa49a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } }
I20260928 12:21:23.034221 4599 raft_consensus.cc:374] T 1a40b4c8b29c458788f28214ddff3177 P a02cd150a0444b2a931e91e89cfa49a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:23.034248 4599 raft_consensus.cc:729] T 1a40b4c8b29c458788f28214ddff3177 P a02cd150a0444b2a931e91e89cfa49a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a02cd150a0444b2a931e91e89cfa49a2, State: Initialized, Role: FOLLOWER
I20260928 12:21:23.034303 4599 consensus_queue.cc:260] T 1a40b4c8b29c458788f28214ddff3177 P a02cd150a0444b2a931e91e89cfa49a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } }
I20260928 12:21:23.034427 4599 ts_tablet_manager.cc:1428] T 1a40b4c8b29c458788f28214ddff3177 P a02cd150a0444b2a931e91e89cfa49a2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:23.034593 4600 tablet_bootstrap.cc:492] T 1a40b4c8b29c458788f28214ddff3177 P dd4cf280ae1748ea8b67e9f08551e3bb: Bootstrap starting.
I20260928 12:21:23.034585 4601 raft_consensus.cc:348] T 1a40b4c8b29c458788f28214ddff3177 P 646a0acf9a8841009d4a3fe5b5e43150 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } }
I20260928 12:21:23.034660 4601 raft_consensus.cc:374] T 1a40b4c8b29c458788f28214ddff3177 P 646a0acf9a8841009d4a3fe5b5e43150 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:23.034682 4601 raft_consensus.cc:729] T 1a40b4c8b29c458788f28214ddff3177 P 646a0acf9a8841009d4a3fe5b5e43150 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 646a0acf9a8841009d4a3fe5b5e43150, State: Initialized, Role: FOLLOWER
I20260928 12:21:23.034721 4601 consensus_queue.cc:260] T 1a40b4c8b29c458788f28214ddff3177 P 646a0acf9a8841009d4a3fe5b5e43150 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } }
I20260928 12:21:23.034842 4601 ts_tablet_manager.cc:1428] T 1a40b4c8b29c458788f28214ddff3177 P 646a0acf9a8841009d4a3fe5b5e43150: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 12:21:23.034971 4600 tablet_bootstrap.cc:654] T 1a40b4c8b29c458788f28214ddff3177 P dd4cf280ae1748ea8b67e9f08551e3bb: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:23.035563 4600 tablet_bootstrap.cc:492] T 1a40b4c8b29c458788f28214ddff3177 P dd4cf280ae1748ea8b67e9f08551e3bb: No bootstrap required, opened a new log
I20260928 12:21:23.035614 4600 ts_tablet_manager.cc:1397] T 1a40b4c8b29c458788f28214ddff3177 P dd4cf280ae1748ea8b67e9f08551e3bb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:21:23.035732 4600 raft_consensus.cc:348] T 1a40b4c8b29c458788f28214ddff3177 P dd4cf280ae1748ea8b67e9f08551e3bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } }
I20260928 12:21:23.035789 4600 raft_consensus.cc:374] T 1a40b4c8b29c458788f28214ddff3177 P dd4cf280ae1748ea8b67e9f08551e3bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:23.035805 4600 raft_consensus.cc:729] T 1a40b4c8b29c458788f28214ddff3177 P dd4cf280ae1748ea8b67e9f08551e3bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd4cf280ae1748ea8b67e9f08551e3bb, State: Initialized, Role: FOLLOWER
I20260928 12:21:23.035845 4600 consensus_queue.cc:260] T 1a40b4c8b29c458788f28214ddff3177 P dd4cf280ae1748ea8b67e9f08551e3bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } }
I20260928 12:21:23.035943 4600 ts_tablet_manager.cc:1428] T 1a40b4c8b29c458788f28214ddff3177 P dd4cf280ae1748ea8b67e9f08551e3bb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:23.177060 4605 raft_consensus.cc:482] T 1a40b4c8b29c458788f28214ddff3177 P 646a0acf9a8841009d4a3fe5b5e43150 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:21:23.177227 4605 raft_consensus.cc:504] T 1a40b4c8b29c458788f28214ddff3177 P 646a0acf9a8841009d4a3fe5b5e43150 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } }
I20260928 12:21:23.177835 4605 leader_election.cc:288] T 1a40b4c8b29c458788f28214ddff3177 P 646a0acf9a8841009d4a3fe5b5e43150 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd4cf280ae1748ea8b67e9f08551e3bb (127.2.56.1:36185), a02cd150a0444b2a931e91e89cfa49a2 (127.2.56.2:46837)
I20260928 12:21:23.181304 4617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:23.178194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:44125 (local address 127.2.56.1:36185)
0928 12:21:23.178321 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:23.178323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:23.178346 (+ 23us) server_negotiation.cc:407] Connection header received
0928 12:21:23.178383 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:23.178384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:23.178394 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:23.178466 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:23.178644 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:23.179236 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:23.180422 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:23.180912 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:23.180984 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:23.180999 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:23.181141 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 12:21:23.181190 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
I20260928 12:21:23.181316 4608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:23.178114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:55633 (local address 127.2.56.2:46837)
0928 12:21:23.178189 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:23.178192 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:23.178298 (+ 106us) server_negotiation.cc:407] Connection header received
0928 12:21:23.178400 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:23.178402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:23.178418 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:23.178464 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:23.178645 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:23.179196 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:23.180461 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:23.180923 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:23.180967 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:23.180979 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:23.181161 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 12:21:23.181212 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 12:21:23.181363 4616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:23.178029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:46837 (local address 127.2.56.3:55633)
0928 12:21:23.178221 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:23.178230 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:23.178298 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:23.178477 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:23.178478 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:23.178481 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:23.178584 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:23.178586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:23.179277 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:23.179278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:23.180344 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:23.180348 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:23.181025 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:23.181026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:23.181086 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:23.181089 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:21:23.181130 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 12:21:23.181183 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":10,"thread_start_us":115,"threads_started":1}
I20260928 12:21:23.181478 4615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:23.178029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:36185 (local address 127.2.56.3:44125)
0928 12:21:23.178207 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:23.178223 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:23.178298 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:23.178492 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:23.178493 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:23.178495 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:23.178579 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:23.178582 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:23.179337 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:23.179338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:23.180346 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:23.180349 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:23.180993 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:23.180994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:23.181074 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:23.181080 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 12:21:23.181127 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 12:21:23.181182 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":8,"thread_start_us":94,"threads_started":1}
I20260928 12:21:23.181792 4160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a40b4c8b29c458788f28214ddff3177" candidate_uuid: "646a0acf9a8841009d4a3fe5b5e43150" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" is_pre_election: true
I20260928 12:21:23.181910 4160 raft_consensus.cc:2454] T 1a40b4c8b29c458788f28214ddff3177 P dd4cf280ae1748ea8b67e9f08551e3bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 646a0acf9a8841009d4a3fe5b5e43150 in term 0.
I20260928 12:21:23.181908 4300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a40b4c8b29c458788f28214ddff3177" candidate_uuid: "646a0acf9a8841009d4a3fe5b5e43150" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a02cd150a0444b2a931e91e89cfa49a2" is_pre_election: true
I20260928 12:21:23.182013 4300 raft_consensus.cc:2454] T 1a40b4c8b29c458788f28214ddff3177 P a02cd150a0444b2a931e91e89cfa49a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 646a0acf9a8841009d4a3fe5b5e43150 in term 0.
I20260928 12:21:23.182122 4372 leader_election.cc:302] T 1a40b4c8b29c458788f28214ddff3177 P 646a0acf9a8841009d4a3fe5b5e43150 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 646a0acf9a8841009d4a3fe5b5e43150, dd4cf280ae1748ea8b67e9f08551e3bb; no voters:
I20260928 12:21:23.182297 4605 raft_consensus.cc:2781] T 1a40b4c8b29c458788f28214ddff3177 P 646a0acf9a8841009d4a3fe5b5e43150 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:21:23.182351 4605 raft_consensus.cc:482] T 1a40b4c8b29c458788f28214ddff3177 P 646a0acf9a8841009d4a3fe5b5e43150 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:21:23.182374 4605 raft_consensus.cc:3037] T 1a40b4c8b29c458788f28214ddff3177 P 646a0acf9a8841009d4a3fe5b5e43150 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:23.183702 4605 raft_consensus.cc:504] T 1a40b4c8b29c458788f28214ddff3177 P 646a0acf9a8841009d4a3fe5b5e43150 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } }
I20260928 12:21:23.183857 4605 leader_election.cc:288] T 1a40b4c8b29c458788f28214ddff3177 P 646a0acf9a8841009d4a3fe5b5e43150 [CANDIDATE]: Term 1 election: Requested vote from peers dd4cf280ae1748ea8b67e9f08551e3bb (127.2.56.1:36185), a02cd150a0444b2a931e91e89cfa49a2 (127.2.56.2:46837)
I20260928 12:21:23.184036 4300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a40b4c8b29c458788f28214ddff3177" candidate_uuid: "646a0acf9a8841009d4a3fe5b5e43150" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a02cd150a0444b2a931e91e89cfa49a2"
I20260928 12:21:23.184064 4160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a40b4c8b29c458788f28214ddff3177" candidate_uuid: "646a0acf9a8841009d4a3fe5b5e43150" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb"
I20260928 12:21:23.184129 4300 raft_consensus.cc:3037] T 1a40b4c8b29c458788f28214ddff3177 P a02cd150a0444b2a931e91e89cfa49a2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:23.184146 4160 raft_consensus.cc:3037] T 1a40b4c8b29c458788f28214ddff3177 P dd4cf280ae1748ea8b67e9f08551e3bb [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:23.184960 4160 raft_consensus.cc:2454] T 1a40b4c8b29c458788f28214ddff3177 P dd4cf280ae1748ea8b67e9f08551e3bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 646a0acf9a8841009d4a3fe5b5e43150 in term 1.
I20260928 12:21:23.185065 4300 raft_consensus.cc:2454] T 1a40b4c8b29c458788f28214ddff3177 P a02cd150a0444b2a931e91e89cfa49a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 646a0acf9a8841009d4a3fe5b5e43150 in term 1.
I20260928 12:21:23.185170 4372 leader_election.cc:302] T 1a40b4c8b29c458788f28214ddff3177 P 646a0acf9a8841009d4a3fe5b5e43150 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 646a0acf9a8841009d4a3fe5b5e43150, dd4cf280ae1748ea8b67e9f08551e3bb; no voters:
I20260928 12:21:23.185292 4605 raft_consensus.cc:2781] T 1a40b4c8b29c458788f28214ddff3177 P 646a0acf9a8841009d4a3fe5b5e43150 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:21:23.185468 4605 raft_consensus.cc:686] T 1a40b4c8b29c458788f28214ddff3177 P 646a0acf9a8841009d4a3fe5b5e43150 [term 1 LEADER]: Becoming Leader. State: Replica: 646a0acf9a8841009d4a3fe5b5e43150, State: Running, Role: LEADER
I20260928 12:21:23.185560 4605 consensus_queue.cc:237] T 1a40b4c8b29c458788f28214ddff3177 P 646a0acf9a8841009d4a3fe5b5e43150 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } }
I20260928 12:21:23.186474 4003 catalog_manager.cc:5495] T 1a40b4c8b29c458788f28214ddff3177 P 646a0acf9a8841009d4a3fe5b5e43150 reported cstate change: term changed from 0 to 1, leader changed from <none> to 646a0acf9a8841009d4a3fe5b5e43150 (127.2.56.3). New cstate: current_term: 1 leader_uuid: "646a0acf9a8841009d4a3fe5b5e43150" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "646a0acf9a8841009d4a3fe5b5e43150" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34425 } health_report { overall_health: HEALTHY } } }
I20260928 12:21:23.222973 4620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:23.220367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48546 (local address 127.2.56.3:34425)
0928 12:21:23.220564 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:23.220565 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:23.220571 (+ 6us) server_negotiation.cc:407] Connection header received
0928 12:21:23.220611 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:23.220611 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:23.220634 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:23.220706 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:21:23.220834 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:23.221454 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:23.222180 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:23.222393 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:23.222441 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:23.222574 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:21:23.222765 (+ 191us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:21:23.222825 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:23.222842 (+ 17us) server_negotiation.cc:299] Negotiation successful
0928 12:21:23.222862 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":128,"threads_started":1}
I20260928 12:21:23.228325 4617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:23.225771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:51261 (local address 127.2.56.1:36185)
0928 12:21:23.225846 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:23.225849 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:23.225956 (+ 107us) server_negotiation.cc:407] Connection header received
0928 12:21:23.226014 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:23.226015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:23.226030 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:23.226083 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:21:23.226225 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:23.226730 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:23.227614 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:23.227834 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:23.227890 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:23.228064 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:21:23.228099 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:21:23.228139 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:23.228196 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 12:21:23.228222 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 12:21:23.228340 4623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:23.225684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:36185 (local address 127.2.56.3:51261)
0928 12:21:23.225885 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:23.225897 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:23.225961 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:23.226086 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:23.226088 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:23.226089 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:21:23.226173 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:23.226175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:23.226799 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:23.226800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:23.227519 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:23.227522 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:23.227919 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:23.227920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:23.227979 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:23.227994 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:21:23.228144 (+ 150us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:21:23.228146 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:21:23.228191 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 12:21:23.228241 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":80,"threads_started":1}
I20260928 12:21:23.230873 4160 raft_consensus.cc:1261] T 1a40b4c8b29c458788f28214ddff3177 P dd4cf280ae1748ea8b67e9f08551e3bb [term 1 FOLLOWER]: Refusing update from remote peer 646a0acf9a8841009d4a3fe5b5e43150: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:21:23.230875 4300 raft_consensus.cc:1261] T 1a40b4c8b29c458788f28214ddff3177 P a02cd150a0444b2a931e91e89cfa49a2 [term 1 FOLLOWER]: Refusing update from remote peer 646a0acf9a8841009d4a3fe5b5e43150: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:21:23.231161 4605 consensus_queue.cc:1035] T 1a40b4c8b29c458788f28214ddff3177 P 646a0acf9a8841009d4a3fe5b5e43150 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd4cf280ae1748ea8b67e9f08551e3bb" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 36185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:21:23.231315 4618 consensus_queue.cc:1035] T 1a40b4c8b29c458788f28214ddff3177 P 646a0acf9a8841009d4a3fe5b5e43150 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a02cd150a0444b2a931e91e89cfa49a2" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:21:23.238433 4620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:23.235899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:50457 (local address 127.2.56.3:34425)
0928 12:21:23.235942 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:23.235944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:23.236061 (+ 117us) server_negotiation.cc:407] Connection header received
0928 12:21:23.236117 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:23.236118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:23.236128 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:23.236184 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:21:23.236320 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:23.236851 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:23.237678 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:23.237995 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:23.238053 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:23.238223 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:21:23.238256 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:21:23.238296 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:23.238351 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 12:21:23.238373 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:21:23.238698 4627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:23.235830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:34425 (local address 127.2.56.1:50457)
0928 12:21:23.235991 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:23.236002 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:23.236066 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:23.236196 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:23.236197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:23.236199 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:21:23.236259 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:23.236261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:23.236925 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:23.236926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:23.237611 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:23.237614 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:23.238089 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:23.238090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:23.238146 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:23.238153 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:21:23.238307 (+ 154us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:21:23.238309 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:21:23.238344 (+ 35us) client_negotiation.cc:240] Negotiation successful
0928 12:21:23.238395 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":116,"threads_started":1}
W20260928 12:21:23.239090 4572 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 12:21:24.758129 4003 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:21:24.758260 4003 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:21:24.759109 4003 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4796d476b91e4f5998165aec0ca20110: Sending DeleteTablet for 3 replicas of tablet 1a40b4c8b29c458788f28214ddff3177
I20260928 12:21:24.759366 4280 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a40b4c8b29c458788f28214ddff3177 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:24 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:33692
I20260928 12:21:24.759366 4140 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a40b4c8b29c458788f28214ddff3177 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:24 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:47094
I20260928 12:21:24.759508 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 4075
I20260928 12:21:24.759569 4420 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a40b4c8b29c458788f28214ddff3177 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:24 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:48542
I20260928 12:21:24.759770 4652 tablet_replica.cc:330] T 1a40b4c8b29c458788f28214ddff3177 P a02cd150a0444b2a931e91e89cfa49a2: stopping tablet replica
I20260928 12:21:24.759848 4652 raft_consensus.cc:2229] T 1a40b4c8b29c458788f28214ddff3177 P a02cd150a0444b2a931e91e89cfa49a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:21:24.759881 4652 raft_consensus.cc:2258] T 1a40b4c8b29c458788f28214ddff3177 P a02cd150a0444b2a931e91e89cfa49a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:24.760336 4652 ts_tablet_manager.cc:1905] T 1a40b4c8b29c458788f28214ddff3177 P a02cd150a0444b2a931e91e89cfa49a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:24.762238 4654 tablet_replica.cc:330] T 1a40b4c8b29c458788f28214ddff3177 P 646a0acf9a8841009d4a3fe5b5e43150: stopping tablet replica
I20260928 12:21:24.762327 4654 raft_consensus.cc:2229] T 1a40b4c8b29c458788f28214ddff3177 P 646a0acf9a8841009d4a3fe5b5e43150 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:21:24.762341 4652 ts_tablet_manager.cc:1918] T 1a40b4c8b29c458788f28214ddff3177 P a02cd150a0444b2a931e91e89cfa49a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 12:21:24.762390 4652 log.cc:1196] T 1a40b4c8b29c458788f28214ddff3177 P a02cd150a0444b2a931e91e89cfa49a2: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-1/wal/wals/1a40b4c8b29c458788f28214ddff3177
I20260928 12:21:24.762460 4654 raft_consensus.cc:2258] T 1a40b4c8b29c458788f28214ddff3177 P 646a0acf9a8841009d4a3fe5b5e43150 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:24.762612 4652 ts_tablet_manager.cc:1939] T 1a40b4c8b29c458788f28214ddff3177 P a02cd150a0444b2a931e91e89cfa49a2: Deleting consensus metadata
I20260928 12:21:24.762734 4654 ts_tablet_manager.cc:1905] T 1a40b4c8b29c458788f28214ddff3177 P 646a0acf9a8841009d4a3fe5b5e43150: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:24.763126 3990 catalog_manager.cc:4841] TS a02cd150a0444b2a931e91e89cfa49a2 (127.2.56.2:46837): tablet 1a40b4c8b29c458788f28214ddff3177 (table test-table [id=5572e1e7f8104707912f32c6854d73ba]) successfully deleted
I20260928 12:21:24.764176 4654 ts_tablet_manager.cc:1918] T 1a40b4c8b29c458788f28214ddff3177 P 646a0acf9a8841009d4a3fe5b5e43150: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 12:21:24.764226 4654 log.cc:1196] T 1a40b4c8b29c458788f28214ddff3177 P 646a0acf9a8841009d4a3fe5b5e43150: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598072638673-2272-0/minicluster-data/ts-2/wal/wals/1a40b4c8b29c458788f28214ddff3177
I20260928 12:21:24.764418 4654 ts_tablet_manager.cc:1939] T 1a40b4c8b29c458788f28214ddff3177 P 646a0acf9a8841009d4a3fe5b5e43150: Deleting consensus metadata
I20260928 12:21:24.764729 3988 catalog_manager.cc:4841] TS 646a0acf9a8841009d4a3fe5b5e43150 (127.2.56.3:34425): tablet 1a40b4c8b29c458788f28214ddff3177 (table test-table [id=5572e1e7f8104707912f32c6854d73ba]) successfully deleted
W20260928 12:21:24.767918 3988 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.56.1:36185 (error 108)
I20260928 12:21:24.768167 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 4215
I20260928 12:21:24.769487 4656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:24.768333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:36185 (local address 127.0.0.1:55210)
0928 12:21:24.769395 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:24.769446 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.1:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1029,"thread_start_us":79,"threads_started":1}
W20260928 12:21:24.769589 3988 catalog_manager.cc:4584] TS dd4cf280ae1748ea8b67e9f08551e3bb (127.2.56.1:36185): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1a40b4c8b29c458788f28214ddff3177: Network error: Client connection negotiation failed: client connection to 127.2.56.1:36185: connect: Connection refused (error 111)
I20260928 12:21:24.773452 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 4355
I20260928 12:21:24.778491 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 3971
2026-09-28T12:21:24Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5774 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:21:24 dist-test-slave-920p krb5kdc[4659](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:21:24 dist-test-slave-920p krb5kdc[4659](info): set up 2 sockets
Sep 28 12:21:24 dist-test-slave-920p krb5kdc[4659](info): commencing operation
krb5kdc: starting...
W20260928 12:21:25.737332 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:21:25 dist-test-slave-920p krb5kdc[4659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:21:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:21:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
I20260928 12:21:25.857239 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:40473
--webserver_interface=127.2.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:36199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:40473
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:25.946609 4672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:25.946856 4672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:25.946879 4672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:25.948129 4672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:21:25.948174 4672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:25.948187 4672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:25.948200 4672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:21:25.948223 4672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:21:25.949792 4672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:36199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:40473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:40473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:25.950093 4672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:25.950335 4672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:25.955782 4678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:25.955792 4677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:25.955770 4680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:25.955902 4672 server_base.cc:1034] running on GCE node
I20260928 12:21:25.956254 4672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:25.956573 4672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:25.957707 4672 hybrid_clock.cc:648] HybridClock initialized: now 1790598085957696 us; error 31 us; skew 500 ppm
Sep 28 12:21:25 dist-test-slave-920p krb5kdc[4659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598085, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:25.967113 4672 init.cc:377] Logged in from keytab as kudu/127.2.56.62@KRBTEST.COM (short username kudu)
I20260928 12:21:25.967672 4672 webserver.cc:460] Webserver started at http://127.2.56.62:38621/ using document root <none> and password file <none>
I20260928 12:21:25.967913 4672 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:25.967964 4672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:25.968106 4672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:25.969411 4672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "62dce27fcf5f442d969d04ac4b3ca5bf"
format_stamp: "Formatted at 2026-09-28 12:21:25 on dist-test-slave-920p"
server_key: "ebb063eda1b3bc640d5817d4c069b9ed"
server_key_iv: "1899595ee08be6f469666e8f8ebd5c4c"
server_key_version: "encryptionkey@0"
I20260928 12:21:25.969795 4672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "62dce27fcf5f442d969d04ac4b3ca5bf"
format_stamp: "Formatted at 2026-09-28 12:21:25 on dist-test-slave-920p"
server_key: "ebb063eda1b3bc640d5817d4c069b9ed"
server_key_iv: "1899595ee08be6f469666e8f8ebd5c4c"
server_key_version: "encryptionkey@0"
I20260928 12:21:25.971143 4672 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:21:25.971828 4687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:25.971982 4672 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:25.972078 4672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "62dce27fcf5f442d969d04ac4b3ca5bf"
format_stamp: "Formatted at 2026-09-28 12:21:25 on dist-test-slave-920p"
server_key: "ebb063eda1b3bc640d5817d4c069b9ed"
server_key_iv: "1899595ee08be6f469666e8f8ebd5c4c"
server_key_version: "encryptionkey@0"
I20260928 12:21:25.972141 4672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:25.983078 4672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:25.984948 4672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:25.985100 4672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:25.989612 4672 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:40473
I20260928 12:21:25.989675 4739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:40473 every 8 connection(s)
I20260928 12:21:25.990096 4672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:21:25.991055 4740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:25.992894 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 4672
I20260928 12:21:25.992995 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:21:25.993203 2272 external_mini_cluster.cc:1428] Setting key c19a49c78b99964e27723dfeea4393c7
I20260928 12:21:25.994112 4740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62dce27fcf5f442d969d04ac4b3ca5bf: Bootstrap starting.
I20260928 12:21:25.994813 4740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62dce27fcf5f442d969d04ac4b3ca5bf: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:25.995117 4740 log.cc:824] T 00000000000000000000000000000000 P 62dce27fcf5f442d969d04ac4b3ca5bf: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:25.995896 4740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62dce27fcf5f442d969d04ac4b3ca5bf: No bootstrap required, opened a new log
I20260928 12:21:25.998332 4740 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62dce27fcf5f442d969d04ac4b3ca5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62dce27fcf5f442d969d04ac4b3ca5bf" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 40473 } }
I20260928 12:21:25.998540 4740 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62dce27fcf5f442d969d04ac4b3ca5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:25.998596 4740 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62dce27fcf5f442d969d04ac4b3ca5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62dce27fcf5f442d969d04ac4b3ca5bf, State: Initialized, Role: FOLLOWER
I20260928 12:21:25.998747 4740 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62dce27fcf5f442d969d04ac4b3ca5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62dce27fcf5f442d969d04ac4b3ca5bf" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 40473 } }
I20260928 12:21:25.998821 4740 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62dce27fcf5f442d969d04ac4b3ca5bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:21:25.998842 4740 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62dce27fcf5f442d969d04ac4b3ca5bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:21:25.998891 4740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62dce27fcf5f442d969d04ac4b3ca5bf [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:25.999634 4740 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62dce27fcf5f442d969d04ac4b3ca5bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62dce27fcf5f442d969d04ac4b3ca5bf" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 40473 } }
I20260928 12:21:25.999733 4740 leader_election.cc:302] T 00000000000000000000000000000000 P 62dce27fcf5f442d969d04ac4b3ca5bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62dce27fcf5f442d969d04ac4b3ca5bf; no voters:
I20260928 12:21:25.999933 4740 leader_election.cc:288] T 00000000000000000000000000000000 P 62dce27fcf5f442d969d04ac4b3ca5bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:21:25.999994 4745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62dce27fcf5f442d969d04ac4b3ca5bf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:21:26.000144 4745 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62dce27fcf5f442d969d04ac4b3ca5bf [term 1 LEADER]: Becoming Leader. State: Replica: 62dce27fcf5f442d969d04ac4b3ca5bf, State: Running, Role: LEADER
I20260928 12:21:26.000301 4740 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62dce27fcf5f442d969d04ac4b3ca5bf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:21:26.000305 4745 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62dce27fcf5f442d969d04ac4b3ca5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62dce27fcf5f442d969d04ac4b3ca5bf" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 40473 } }
I20260928 12:21:26.000705 4745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62dce27fcf5f442d969d04ac4b3ca5bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62dce27fcf5f442d969d04ac4b3ca5bf. Latest consensus state: current_term: 1 leader_uuid: "62dce27fcf5f442d969d04ac4b3ca5bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62dce27fcf5f442d969d04ac4b3ca5bf" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 40473 } } }
I20260928 12:21:26.000775 4745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62dce27fcf5f442d969d04ac4b3ca5bf [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:26.000931 4746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62dce27fcf5f442d969d04ac4b3ca5bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62dce27fcf5f442d969d04ac4b3ca5bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62dce27fcf5f442d969d04ac4b3ca5bf" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 40473 } } }
I20260928 12:21:26.000998 4746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62dce27fcf5f442d969d04ac4b3ca5bf [sys.catalog]: This master's current role is: LEADER
Sep 28 12:21:25 dist-test-slave-920p krb5kdc[4659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:26.001125 4752 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:21:26.001953 4752 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:21:26.003711 4752 catalog_manager.cc:1347] Generated new cluster ID: 68073f030cfe4d6eb2f274b2c2e8cc3f
I20260928 12:21:26.003763 4752 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:21:26.005317 4743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:25.993816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43826 (local address 127.2.56.62:40473)
0928 12:21:25.994025 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:25.994036 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:25.994064 (+ 28us) server_negotiation.cc:407] Connection header received
0928 12:21:25.994617 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:25.994626 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:25.994685 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:25.994833 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:25.995567 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:25.997023 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:25.997870 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:25.998526 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:25.998645 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:26.001797 (+ 3152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:26.001802 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:26.001803 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:26.001818 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:26.003850 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:26.004187 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:26.004191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:26.004191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:26.004280 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:26.004456 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:26.004457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:26.004458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:26.004709 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:26.004834 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:26.004954 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 12:21:26.005091 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":98,"threads_started":1}
I20260928 12:21:26.025153 4752 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:21:26.025795 4752 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:21:26.035962 4752 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62dce27fcf5f442d969d04ac4b3ca5bf: Generated new TSK 0
I20260928 12:21:26.036177 4752 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab.
Entry for principal kudu/127.2.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab.
I20260928 12:21:26.065997 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.1:0
--local_ip_for_outbound_sockets=127.2.56.1
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:40473
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:36199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:26.153702 4766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:26.153932 4766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:26.153954 4766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:26.155143 4766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:26.155177 4766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:26.155202 4766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.1
I20260928 12:21:26.156634 4766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:36199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:40473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.1
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:26.156857 4766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:26.157083 4766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:26.162057 4771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:26.162114 4772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:26.162062 4774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:26.162325 4766 server_base.cc:1034] running on GCE node
I20260928 12:21:26.162551 4766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:26.162782 4766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:26.163923 4766 hybrid_clock.cc:648] HybridClock initialized: now 1790598086163910 us; error 29 us; skew 500 ppm
Sep 28 12:21:26 dist-test-slave-920p krb5kdc[4659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598086, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:26.166558 4766 init.cc:377] Logged in from keytab as kudu/127.2.56.1@KRBTEST.COM (short username kudu)
I20260928 12:21:26.166985 4766 webserver.cc:460] Webserver started at http://127.2.56.1:38087/ using document root <none> and password file <none>
I20260928 12:21:26.167210 4766 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:26.167258 4766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:26.167387 4766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:26.168499 4766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-0/data/instance:
uuid: "373007b551a94c30a49fc9e552ae9b20"
format_stamp: "Formatted at 2026-09-28 12:21:26 on dist-test-slave-920p"
server_key: "48e4ef898e91966c365dad3630f9fd20"
server_key_iv: "1070cf2197831cb2b28cdbed20a41f95"
server_key_version: "encryptionkey@0"
I20260928 12:21:26.168865 4766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance:
uuid: "373007b551a94c30a49fc9e552ae9b20"
format_stamp: "Formatted at 2026-09-28 12:21:26 on dist-test-slave-920p"
server_key: "48e4ef898e91966c365dad3630f9fd20"
server_key_iv: "1070cf2197831cb2b28cdbed20a41f95"
server_key_version: "encryptionkey@0"
I20260928 12:21:26.170285 4766 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:21:26.170957 4781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:26.171104 4766 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:26.171185 4766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-0/wal
uuid: "373007b551a94c30a49fc9e552ae9b20"
format_stamp: "Formatted at 2026-09-28 12:21:26 on dist-test-slave-920p"
server_key: "48e4ef898e91966c365dad3630f9fd20"
server_key_iv: "1070cf2197831cb2b28cdbed20a41f95"
server_key_version: "encryptionkey@0"
I20260928 12:21:26.171252 4766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:26.200374 4766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:26.203042 4766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:26.203205 4766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:26.203485 4766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:21:26.203833 4766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:26.203867 4766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:26.203900 4766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:26.203922 4766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:26.210403 4766 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.1:38589
I20260928 12:21:26.210467 4894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.1:38589 every 8 connection(s)
I20260928 12:21:26.210868 4766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
I20260928 12:21:26.211275 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 4766
I20260928 12:21:26.211385 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance
I20260928 12:21:26.211582 2272 external_mini_cluster.cc:1428] Setting key 62cec5a3a4bbbc461c77871c1ad3d70a
Sep 28 12:21:26 dist-test-slave-920p krb5kdc[4659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598086, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.56.2@KRBTEST.COM; defaulting to no policy
I20260928 12:21:26.221427 4743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:26.211738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:49999 (local address 127.2.56.62:40473)
0928 12:21:26.211797 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:26.211799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:26.212014 (+ 215us) server_negotiation.cc:407] Connection header received
0928 12:21:26.212591 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:26.212592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:26.212606 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:26.212653 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:26.213775 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:26.214336 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:26.215440 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:26.215677 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:26.215734 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:26.218551 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:26.218552 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:26.218553 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:26.218561 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:26.219968 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:26.220451 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:26.220452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:26.220453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:26.220509 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:26.220732 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:26.220732 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:26.220733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:26.220858 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:26.220938 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:26.221203 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 12:21:26.221341 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:21:26.221613 4897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:26.211700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:40473 (local address 127.2.56.1:49999)
0928 12:21:26.211869 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:26.211919 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:26.212476 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:26.212743 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:26.212749 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:26.213425 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:26.213681 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:26.213687 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:26.214423 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:26.214424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:26.215360 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:26.215362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:26.215750 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:26.215750 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:26.215898 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:26.216530 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:26.216540 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:26.218437 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:26.220063 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:26.220074 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:26.220077 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:26.220385 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:26.220562 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:26.220563 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:26.220563 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:26.220687 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:26.220905 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:26.220910 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:26.221082 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 12:21:26.221172 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 12:21:26.221368 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
I20260928 12:21:26.222137 4895 heartbeater.cc:347] Connected to a master server at 127.2.56.62:40473
I20260928 12:21:26.222254 4895 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:26.222507 4895 heartbeater.cc:511] Master 127.2.56.62:40473 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.2@KRBTEST.COM" created.
I20260928 12:21:26.223104 4704 ts_manager.cc:194] Registered new tserver with Master: 373007b551a94c30a49fc9e552ae9b20 (127.2.56.1:38589)
I20260928 12:21:26.223774 4704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.1@KRBTEST.COM'} at 127.2.56.1:49999
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab.
Entry for principal kudu/127.2.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab.
I20260928 12:21:26.237540 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.2:0
--local_ip_for_outbound_sockets=127.2.56.2
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:40473
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:36199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:26.324467 4900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:26.324702 4900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:26.324724 4900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:26.326009 4900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:26.326058 4900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:26.326085 4900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.2
I20260928 12:21:26.327543 4900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:36199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:40473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.2
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:26.327780 4900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:26.328003 4900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:26.333365 4906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:26.333391 4900 server_base.cc:1034] running on GCE node
W20260928 12:21:26.333520 4908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:26.333596 4905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:26.333847 4900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:26.334152 4900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:26.335322 4900 hybrid_clock.cc:648] HybridClock initialized: now 1790598086335289 us; error 49 us; skew 500 ppm
Sep 28 12:21:26 dist-test-slave-920p krb5kdc[4659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598086, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:26.338558 4900 init.cc:377] Logged in from keytab as kudu/127.2.56.2@KRBTEST.COM (short username kudu)
I20260928 12:21:26.339126 4900 webserver.cc:460] Webserver started at http://127.2.56.2:40625/ using document root <none> and password file <none>
I20260928 12:21:26.339362 4900 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:26.339408 4900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:26.339525 4900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:26.340799 4900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-1/data/instance:
uuid: "6ddeb6cbdbc24b7d826b1d964e2d6420"
format_stamp: "Formatted at 2026-09-28 12:21:26 on dist-test-slave-920p"
server_key: "69bec45069c1e8ee1cab887ac0b4278e"
server_key_iv: "74955f9cf010ce522b8af4a69cd7a256"
server_key_version: "encryptionkey@0"
I20260928 12:21:26.341185 4900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance:
uuid: "6ddeb6cbdbc24b7d826b1d964e2d6420"
format_stamp: "Formatted at 2026-09-28 12:21:26 on dist-test-slave-920p"
server_key: "69bec45069c1e8ee1cab887ac0b4278e"
server_key_iv: "74955f9cf010ce522b8af4a69cd7a256"
server_key_version: "encryptionkey@0"
I20260928 12:21:26.342850 4900 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:21:26.343711 4915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:26.343909 4900 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:21:26.344023 4900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-1/wal
uuid: "6ddeb6cbdbc24b7d826b1d964e2d6420"
format_stamp: "Formatted at 2026-09-28 12:21:26 on dist-test-slave-920p"
server_key: "69bec45069c1e8ee1cab887ac0b4278e"
server_key_iv: "74955f9cf010ce522b8af4a69cd7a256"
server_key_version: "encryptionkey@0"
I20260928 12:21:26.344111 4900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:26.359200 4900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:26.361703 4900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:26.361850 4900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:26.362134 4900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:21:26.362505 4900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:26.362541 4900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:26.362574 4900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:26.362597 4900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:26.369629 4900 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.2:34349
I20260928 12:21:26.369673 5028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.2:34349 every 8 connection(s)
I20260928 12:21:26.370146 4900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
I20260928 12:21:26.373019 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 4900
I20260928 12:21:26.373137 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance
I20260928 12:21:26.373440 2272 external_mini_cluster.cc:1428] Setting key 4394ee7a43ebc2c43681a250ea9e0da4
Sep 28 12:21:26 dist-test-slave-920p krb5kdc[4659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598086, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:26.382306 4743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:26.371151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:38705 (local address 127.2.56.62:40473)
0928 12:21:26.371240 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:26.371243 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:26.371408 (+ 165us) server_negotiation.cc:407] Connection header received
0928 12:21:26.372043 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:26.372044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:26.372058 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:26.372114 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:26.373580 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:26.374218 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:26.375699 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:26.376014 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:26.376089 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:26.378748 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:26.378751 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:26.378751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:26.378761 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:26.380301 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:26.380895 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:26.380896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:26.380896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:26.380954 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:26.381376 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:26.381377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:26.381378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:26.381559 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:26.381639 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:26.382088 (+ 449us) server_negotiation.cc:299] Negotiation successful
0928 12:21:26.382208 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 12:21:26.382642 5031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:26.371070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:40473 (local address 127.2.56.2:38705)
0928 12:21:26.371255 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:26.371305 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:26.371936 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:26.372198 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:26.372203 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:26.373001 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:26.373447 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:26.373459 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:26.374335 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:26.374338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:26.375577 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:26.375581 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:26.376124 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:26.376125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:26.376278 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:26.376906 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:26.376919 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:26.378630 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:26.380402 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:26.380415 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:26.380420 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:26.380811 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:26.381016 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:26.381017 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:26.381017 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:26.381195 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:26.381643 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:26.381653 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:26.381905 (+ 252us) client_negotiation.cc:769] Sending connection context
0928 12:21:26.382050 (+ 145us) client_negotiation.cc:240] Negotiation successful
0928 12:21:26.382296 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for kudu/127.2.56.3@KRBTEST.COM; defaulting to no policy
I20260928 12:21:26.383345 5029 heartbeater.cc:347] Connected to a master server at 127.2.56.62:40473
I20260928 12:21:26.383498 5029 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:26.383752 5029 heartbeater.cc:511] Master 127.2.56.62:40473 requested a full tablet report, sending...
I20260928 12:21:26.384270 4704 ts_manager.cc:194] Registered new tserver with Master: 6ddeb6cbdbc24b7d826b1d964e2d6420 (127.2.56.2:34349)
I20260928 12:21:26.384757 4704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.2@KRBTEST.COM'} at 127.2.56.2:38705
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab.
Entry for principal kudu/127.2.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab.
I20260928 12:21:26.402138 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.3:0
--local_ip_for_outbound_sockets=127.2.56.3
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:40473
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:36199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:26.490784 5034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:26.491022 5034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:26.491046 5034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:26.492255 5034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:26.492300 5034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:26.492331 5034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.3
I20260928 12:21:26.493834 5034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:36199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:40473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.3
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:26.494086 5034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:26.494315 5034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:26.500099 5040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:26.500010 5034 server_base.cc:1034] running on GCE node
W20260928 12:21:26.500072 5039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:26.500036 5042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:26.500515 5034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:26.500800 5034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:26.501968 5034 hybrid_clock.cc:648] HybridClock initialized: now 1790598086501951 us; error 43 us; skew 500 ppm
Sep 28 12:21:26 dist-test-slave-920p krb5kdc[4659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598086, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:26.504731 5034 init.cc:377] Logged in from keytab as kudu/127.2.56.3@KRBTEST.COM (short username kudu)
I20260928 12:21:26.505209 5034 webserver.cc:460] Webserver started at http://127.2.56.3:43193/ using document root <none> and password file <none>
I20260928 12:21:26.505465 5034 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:26.505518 5034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:26.505656 5034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:26.506871 5034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-2/data/instance:
uuid: "310d890e49b2437ab2f96b999d15510e"
format_stamp: "Formatted at 2026-09-28 12:21:26 on dist-test-slave-920p"
server_key: "f8539d3ac324ec326073cb5c6ee32b21"
server_key_iv: "6852bdbdfac749f149070b7b06d36789"
server_key_version: "encryptionkey@0"
I20260928 12:21:26.507284 5034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance:
uuid: "310d890e49b2437ab2f96b999d15510e"
format_stamp: "Formatted at 2026-09-28 12:21:26 on dist-test-slave-920p"
server_key: "f8539d3ac324ec326073cb5c6ee32b21"
server_key_iv: "6852bdbdfac749f149070b7b06d36789"
server_key_version: "encryptionkey@0"
I20260928 12:21:26.508656 5034 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:21:26.509370 5049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:26.509521 5034 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:26.509610 5034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-2/wal
uuid: "310d890e49b2437ab2f96b999d15510e"
format_stamp: "Formatted at 2026-09-28 12:21:26 on dist-test-slave-920p"
server_key: "f8539d3ac324ec326073cb5c6ee32b21"
server_key_iv: "6852bdbdfac749f149070b7b06d36789"
server_key_version: "encryptionkey@0"
I20260928 12:21:26.509675 5034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:26.523958 5034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:26.526475 5034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:26.526650 5034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:26.526913 5034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:21:26.527256 5034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:26.527290 5034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:26.527321 5034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:26.527343 5034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:26.533756 5034 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.3:42745
I20260928 12:21:26.533804 5162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.3:42745 every 8 connection(s)
I20260928 12:21:26.534168 5034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
I20260928 12:21:26.537698 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 5034
I20260928 12:21:26.537809 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance
I20260928 12:21:26.538000 2272 external_mini_cluster.cc:1428] Setting key d279b710e90ec6184a59e17644c9010b
Sep 28 12:21:26 dist-test-slave-920p krb5kdc[4659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598086, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:26.543851 4743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:26.534957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:37279 (local address 127.2.56.62:40473)
0928 12:21:26.535042 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:26.535045 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:26.535228 (+ 183us) server_negotiation.cc:407] Connection header received
0928 12:21:26.535823 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:26.535825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:26.535841 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:26.535923 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:26.537063 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:26.537773 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:26.539086 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:26.539295 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:26.539365 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:26.541556 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:26.541558 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:26.541558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:26.541565 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:26.542690 (+ 1125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:26.543034 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:26.543035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:26.543035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:26.543081 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:26.543257 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:26.543258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:26.543258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:26.543372 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:26.543418 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:26.543658 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 12:21:26.543723 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 12:21:26.543985 5165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:26.534922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:40473 (local address 127.2.56.3:37279)
0928 12:21:26.535092 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:26.535134 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:26.535700 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:26.536023 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:26.536028 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:26.536712 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:26.536971 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:26.536977 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:26.537897 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:26.537899 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:26.539023 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:26.539026 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:26.539353 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:26.539354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:26.539474 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:26.540007 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:26.540016 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:26.541473 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:26.542753 (+ 1280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:26.542759 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:26.542761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:26.542981 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:26.543124 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:26.543124 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:26.543125 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:26.543218 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:26.543413 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:26.543417 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:26.543555 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 12:21:26.543640 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 12:21:26.543775 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":92,"threads_started":1}
I20260928 12:21:26.544605 5163 heartbeater.cc:347] Connected to a master server at 127.2.56.62:40473
I20260928 12:21:26.544713 5163 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:26.544939 5163 heartbeater.cc:511] Master 127.2.56.62:40473 requested a full tablet report, sending...
I20260928 12:21:26.545435 4704 ts_manager.cc:194] Registered new tserver with Master: 310d890e49b2437ab2f96b999d15510e (127.2.56.3:42745)
I20260928 12:21:26.545899 4704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.3@KRBTEST.COM'} at 127.2.56.3:37279
I20260928 12:21:26.549851 2272 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 12:21:26.561517 4743 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:21:26.560508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43832 (local address 127.2.56.62:40473)
0928 12:21:26.560575 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:26.560577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:26.560659 (+ 82us) server_negotiation.cc:407] Connection header received
0928 12:21:26.560722 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:26.560723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:26.560734 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:26.560792 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:26.561424 (+ 632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43832: BlockingRecv error: recv got EOF from 127.0.0.1:43832 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 12:21:26.563113 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 4766
I20260928 12:21:26.567826 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 4900
I20260928 12:21:26.572149 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 5034
I20260928 12:21:26.576236 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 4672
2026-09-28T12:21:26Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1796 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:21:26 dist-test-slave-920p krb5kdc[5175](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:21:26 dist-test-slave-920p krb5kdc[5175](info): set up 2 sockets
Sep 28 12:21:26 dist-test-slave-920p krb5kdc[5175](info): commencing operation
krb5kdc: starting...
W20260928 12:21:27.567699 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260928 12:21:27.778241 2272 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.210s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260928 12:21:28.121855 2272 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.276s user 0.001s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598072638673-2272-0/krb5kdc/test-user.keytab.
W20260928 12:21:28.915997 2272 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.794s user 0.001s sys 0.001s
Sep 28 12:21:28 dist-test-slave-920p krb5kdc[5175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:21:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:21:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
W20260928 12:21:30.149115 2272 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.2.56.62: real 1.146s user 0.001s sys 0.002s
I20260928 12:21:30.149567 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:34515
--webserver_interface=127.2.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:32807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:34515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:30.239178 5188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:30.239411 5188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:30.239432 5188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:30.240633 5188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:21:30.240674 5188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:30.240685 5188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:30.240696 5188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:21:30.240707 5188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:21:30.242384 5188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:32807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:34515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:34515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:30.242647 5188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:30.242875 5188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:30.247756 5194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:30.247828 5193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:30.247780 5196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:30.248090 5188 server_base.cc:1034] running on GCE node
I20260928 12:21:30.248350 5188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:30.248673 5188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:30.249819 5188 hybrid_clock.cc:648] HybridClock initialized: now 1790598090249802 us; error 33 us; skew 500 ppm
Sep 28 12:21:30 dist-test-slave-920p krb5kdc[5175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598090, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:30.252404 5188 init.cc:377] Logged in from keytab as kudu/127.2.56.62@KRBTEST.COM (short username kudu)
I20260928 12:21:30.252831 5188 webserver.cc:460] Webserver started at http://127.2.56.62:44363/ using document root <none> and password file <none>
I20260928 12:21:30.253047 5188 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:30.253091 5188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:30.253216 5188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:30.254388 5188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "30dc0d9f3694499c8f8f560922173fa5"
format_stamp: "Formatted at 2026-09-28 12:21:30 on dist-test-slave-920p"
server_key: "d3a72edc97e5cc1efcfebac029c3270a"
server_key_iv: "3e1ad3d2f224c259804eeeeaafae5fb2"
server_key_version: "encryptionkey@0"
I20260928 12:21:30.254757 5188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "30dc0d9f3694499c8f8f560922173fa5"
format_stamp: "Formatted at 2026-09-28 12:21:30 on dist-test-slave-920p"
server_key: "d3a72edc97e5cc1efcfebac029c3270a"
server_key_iv: "3e1ad3d2f224c259804eeeeaafae5fb2"
server_key_version: "encryptionkey@0"
I20260928 12:21:30.256029 5188 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:21:30.256661 5203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:30.256801 5188 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:30.256893 5188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "30dc0d9f3694499c8f8f560922173fa5"
format_stamp: "Formatted at 2026-09-28 12:21:30 on dist-test-slave-920p"
server_key: "d3a72edc97e5cc1efcfebac029c3270a"
server_key_iv: "3e1ad3d2f224c259804eeeeaafae5fb2"
server_key_version: "encryptionkey@0"
I20260928 12:21:30.256953 5188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:30.283232 5188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:30.285709 5188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:30.285861 5188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:30.290369 5188 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:34515
I20260928 12:21:30.290409 5255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:34515 every 8 connection(s)
I20260928 12:21:30.290796 5188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:21:30.291744 5256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:30.294405 5256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30dc0d9f3694499c8f8f560922173fa5: Bootstrap starting.
I20260928 12:21:30.295053 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 5188
I20260928 12:21:30.295091 5256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30dc0d9f3694499c8f8f560922173fa5: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:30.295132 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:21:30.295296 2272 external_mini_cluster.cc:1428] Setting key f98d04f6bdcfe634d6d490ea03e90d20
I20260928 12:21:30.295390 5256 log.cc:824] T 00000000000000000000000000000000 P 30dc0d9f3694499c8f8f560922173fa5: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:30.296176 5256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30dc0d9f3694499c8f8f560922173fa5: No bootstrap required, opened a new log
I20260928 12:21:30.298455 5256 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30dc0d9f3694499c8f8f560922173fa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30dc0d9f3694499c8f8f560922173fa5" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34515 } }
I20260928 12:21:30.298637 5256 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30dc0d9f3694499c8f8f560922173fa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:30.298684 5256 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30dc0d9f3694499c8f8f560922173fa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30dc0d9f3694499c8f8f560922173fa5, State: Initialized, Role: FOLLOWER
I20260928 12:21:30.298820 5256 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30dc0d9f3694499c8f8f560922173fa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30dc0d9f3694499c8f8f560922173fa5" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34515 } }
I20260928 12:21:30.298910 5256 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30dc0d9f3694499c8f8f560922173fa5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:21:30.298949 5256 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30dc0d9f3694499c8f8f560922173fa5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:21:30.298995 5256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30dc0d9f3694499c8f8f560922173fa5 [term 0 FOLLOWER]: Advancing to term 1
Sep 28 12:21:30 dist-test-slave-920p krb5kdc[5175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:30.305240 5260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:30.295783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58262 (local address 127.2.56.62:34515)
0928 12:21:30.295956 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:30.295965 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:30.295993 (+ 28us) server_negotiation.cc:407] Connection header received
0928 12:21:30.296477 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:30.296485 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:30.296541 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:30.296683 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:30.297501 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:30.298438 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:30.299527 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:30.300020 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:30.300104 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:30.302532 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:30.302537 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:30.302537 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:30.302549 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:30.303930 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:30.304245 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:30.304248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:30.304249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:30.304331 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:30.304465 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:30.304466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:30.304466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:30.304709 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:30.304816 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:30.304889 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 12:21:30.305018 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
I20260928 12:21:30.306339 2272 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:21:30.320427 5260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:30.317165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58274 (local address 127.2.56.62:34515)
0928 12:21:30.317217 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:30.317219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:30.317361 (+ 142us) server_negotiation.cc:407] Connection header received
0928 12:21:30.317417 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:30.317418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:30.317428 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:30.317484 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:30.317840 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:30.318370 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:30.319079 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:30.319276 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:30.319322 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:30.319971 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:30.319972 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:30.319973 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:21:30.320211 (+ 238us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:30.320231 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:21:30.320237 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:30.320279 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:30.320329 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 12:21:30.320353 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 12:21:30.348990 5256 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30dc0d9f3694499c8f8f560922173fa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30dc0d9f3694499c8f8f560922173fa5" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34515 } }
I20260928 12:21:30.349238 5256 leader_election.cc:302] T 00000000000000000000000000000000 P 30dc0d9f3694499c8f8f560922173fa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30dc0d9f3694499c8f8f560922173fa5; no voters:
I20260928 12:21:30.349565 5256 leader_election.cc:288] T 00000000000000000000000000000000 P 30dc0d9f3694499c8f8f560922173fa5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:21:30.349628 5268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30dc0d9f3694499c8f8f560922173fa5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:21:30.349821 5268 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30dc0d9f3694499c8f8f560922173fa5 [term 1 LEADER]: Becoming Leader. State: Replica: 30dc0d9f3694499c8f8f560922173fa5, State: Running, Role: LEADER
I20260928 12:21:30.349891 5256 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30dc0d9f3694499c8f8f560922173fa5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:21:30.350003 5268 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30dc0d9f3694499c8f8f560922173fa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30dc0d9f3694499c8f8f560922173fa5" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34515 } }
I20260928 12:21:30.350463 5269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30dc0d9f3694499c8f8f560922173fa5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30dc0d9f3694499c8f8f560922173fa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30dc0d9f3694499c8f8f560922173fa5" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34515 } } }
I20260928 12:21:30.350555 5269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30dc0d9f3694499c8f8f560922173fa5 [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:30.350832 5270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30dc0d9f3694499c8f8f560922173fa5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30dc0d9f3694499c8f8f560922173fa5. Latest consensus state: current_term: 1 leader_uuid: "30dc0d9f3694499c8f8f560922173fa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30dc0d9f3694499c8f8f560922173fa5" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34515 } } }
I20260928 12:21:30.351226 5270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30dc0d9f3694499c8f8f560922173fa5 [sys.catalog]: This master's current role is: LEADER
W20260928 12:21:30.352011 5284 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 30dc0d9f3694499c8f8f560922173fa5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:21:30.352283 5284 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:21:30.352334 5277 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:21:30.352516 5277 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:21:30.354256 5277 catalog_manager.cc:1347] Generated new cluster ID: 3682f26a828b4bd9bb7ccd32cad124dc
I20260928 12:21:30.354310 5277 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:21:30.371235 5277 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:21:30.371948 5277 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:21:30.374323 5277 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30dc0d9f3694499c8f8f560922173fa5: Generated new TSK 0
I20260928 12:21:30.374511 5277 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:21:30.384310 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 5188
2026-09-28T12:21:30Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (3806 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:21:30 dist-test-slave-920p krb5kdc[5289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:21:30 dist-test-slave-920p krb5kdc[5289](info): set up 2 sockets
Sep 28 12:21:30 dist-test-slave-920p krb5kdc[5289](info): commencing operation
krb5kdc: starting...
W20260928 12:21:31.641494 2272 mini_kdc.cc:121] Time spent starting KDC: real 1.238s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:21:31 dist-test-slave-920p krb5kdc[5289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:21:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:21:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
I20260928 12:21:31.760887 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:40853
--webserver_interface=127.2.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:39533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:40853
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:31.847325 5302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:31.847558 5302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:31.847580 5302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:31.848780 5302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:21:31.848824 5302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:31.848836 5302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:31.848847 5302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:21:31.848858 5302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:21:31.850394 5302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:39533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:40853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:40853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:31.850651 5302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:31.850879 5302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:31.856073 5310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:31.856122 5308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:31.856122 5307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:31.856344 5302 server_base.cc:1034] running on GCE node
I20260928 12:21:31.856602 5302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:31.856914 5302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:31.858055 5302 hybrid_clock.cc:648] HybridClock initialized: now 1790598091858045 us; error 37 us; skew 500 ppm
Sep 28 12:21:31 dist-test-slave-920p krb5kdc[5289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598091, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:31.870046 5302 init.cc:377] Logged in from keytab as kudu/127.2.56.62@KRBTEST.COM (short username kudu)
I20260928 12:21:31.870589 5302 webserver.cc:460] Webserver started at http://127.2.56.62:44605/ using document root <none> and password file <none>
I20260928 12:21:31.870846 5302 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:31.870898 5302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:31.871038 5302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:31.872241 5302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "2b6dc5ed31e64b87b508d2b66d6bafe2"
format_stamp: "Formatted at 2026-09-28 12:21:31 on dist-test-slave-920p"
server_key: "2ed152401ffb8a10c43899925f07aa12"
server_key_iv: "c989fa73f2a89433e26e996620dc2589"
server_key_version: "encryptionkey@0"
I20260928 12:21:31.872603 5302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "2b6dc5ed31e64b87b508d2b66d6bafe2"
format_stamp: "Formatted at 2026-09-28 12:21:31 on dist-test-slave-920p"
server_key: "2ed152401ffb8a10c43899925f07aa12"
server_key_iv: "c989fa73f2a89433e26e996620dc2589"
server_key_version: "encryptionkey@0"
I20260928 12:21:31.874035 5302 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:21:31.874747 5317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:31.874898 5302 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:31.874994 5302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "2b6dc5ed31e64b87b508d2b66d6bafe2"
format_stamp: "Formatted at 2026-09-28 12:21:31 on dist-test-slave-920p"
server_key: "2ed152401ffb8a10c43899925f07aa12"
server_key_iv: "c989fa73f2a89433e26e996620dc2589"
server_key_version: "encryptionkey@0"
I20260928 12:21:31.875056 5302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:31.888437 5302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:31.890545 5302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:31.890683 5302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:31.895079 5302 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:40853
I20260928 12:21:31.895112 5369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:40853 every 8 connection(s)
I20260928 12:21:31.895490 5302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:21:31.896217 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 5302
I20260928 12:21:31.896334 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:21:31.896564 2272 external_mini_cluster.cc:1428] Setting key 04fb786a35d1a03aee12b3b8752d8038
I20260928 12:21:31.896775 5370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:31.900352 5370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b6dc5ed31e64b87b508d2b66d6bafe2: Bootstrap starting.
I20260928 12:21:31.901139 5370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b6dc5ed31e64b87b508d2b66d6bafe2: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:31.901466 5370 log.cc:824] T 00000000000000000000000000000000 P 2b6dc5ed31e64b87b508d2b66d6bafe2: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:31.902254 5370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b6dc5ed31e64b87b508d2b66d6bafe2: No bootstrap required, opened a new log
Sep 28 12:21:31 dist-test-slave-920p krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:31.904666 5370 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b6dc5ed31e64b87b508d2b66d6bafe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b6dc5ed31e64b87b508d2b66d6bafe2" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 40853 } }
I20260928 12:21:31.904830 5370 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b6dc5ed31e64b87b508d2b66d6bafe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:31.904873 5370 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b6dc5ed31e64b87b508d2b66d6bafe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b6dc5ed31e64b87b508d2b66d6bafe2, State: Initialized, Role: FOLLOWER
I20260928 12:21:31.905015 5370 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b6dc5ed31e64b87b508d2b66d6bafe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b6dc5ed31e64b87b508d2b66d6bafe2" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 40853 } }
I20260928 12:21:31.905085 5370 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b6dc5ed31e64b87b508d2b66d6bafe2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:21:31.905126 5370 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b6dc5ed31e64b87b508d2b66d6bafe2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:21:31.905175 5370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b6dc5ed31e64b87b508d2b66d6bafe2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:31.905799 5370 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b6dc5ed31e64b87b508d2b66d6bafe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b6dc5ed31e64b87b508d2b66d6bafe2" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 40853 } }
I20260928 12:21:31.905915 5370 leader_election.cc:302] T 00000000000000000000000000000000 P 2b6dc5ed31e64b87b508d2b66d6bafe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b6dc5ed31e64b87b508d2b66d6bafe2; no voters:
I20260928 12:21:31.906087 5370 leader_election.cc:288] T 00000000000000000000000000000000 P 2b6dc5ed31e64b87b508d2b66d6bafe2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:21:31.906131 5375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b6dc5ed31e64b87b508d2b66d6bafe2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:21:31.906304 5375 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b6dc5ed31e64b87b508d2b66d6bafe2 [term 1 LEADER]: Becoming Leader. State: Replica: 2b6dc5ed31e64b87b508d2b66d6bafe2, State: Running, Role: LEADER
I20260928 12:21:31.906419 5370 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b6dc5ed31e64b87b508d2b66d6bafe2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:21:31.906404 5375 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b6dc5ed31e64b87b508d2b66d6bafe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b6dc5ed31e64b87b508d2b66d6bafe2" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 40853 } }
I20260928 12:21:31.906872 5376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b6dc5ed31e64b87b508d2b66d6bafe2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b6dc5ed31e64b87b508d2b66d6bafe2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b6dc5ed31e64b87b508d2b66d6bafe2" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 40853 } } }
I20260928 12:21:31.906961 5376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b6dc5ed31e64b87b508d2b66d6bafe2 [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:31.907186 5375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b6dc5ed31e64b87b508d2b66d6bafe2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b6dc5ed31e64b87b508d2b66d6bafe2. Latest consensus state: current_term: 1 leader_uuid: "2b6dc5ed31e64b87b508d2b66d6bafe2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b6dc5ed31e64b87b508d2b66d6bafe2" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 40853 } } }
I20260928 12:21:31.907509 5375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b6dc5ed31e64b87b508d2b66d6bafe2 [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:31.907756 5383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:21:31.908339 5373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:31.897147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56822 (local address 127.2.56.62:40853)
0928 12:21:31.897346 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:31.897357 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:31.897385 (+ 28us) server_negotiation.cc:407] Connection header received
0928 12:21:31.898053 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:31.898061 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:31.898119 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:31.898252 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:31.899049 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:31.900007 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:31.901063 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:31.901627 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:31.901730 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:31.904434 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:31.904439 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:31.904440 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:31.904454 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:31.906266 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:31.906720 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:31.906723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:31.906724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:31.906836 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:31.907068 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:31.907069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:31.907069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:31.907401 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:31.907567 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:31.907960 (+ 393us) server_negotiation.cc:299] Negotiation successful
0928 12:21:31.908121 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":81,"threads_started":1}
I20260928 12:21:31.908401 5383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:21:31.910054 5383 catalog_manager.cc:1347] Generated new cluster ID: c1902d42cc4843668b7baad4d9456a51
I20260928 12:21:31.910102 5383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:21:31.923094 5383 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:21:31.923666 5383 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:21:31.930251 5383 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b6dc5ed31e64b87b508d2b66d6bafe2: Generated new TSK 0
I20260928 12:21:31.930500 5383 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab.
Entry for principal kudu/127.2.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab.
I20260928 12:21:31.960575 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.1:0
--local_ip_for_outbound_sockets=127.2.56.1
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:40853
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:39533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:32.048339 5396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:32.048612 5396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:32.048635 5396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:32.049924 5396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:32.049976 5396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:32.050004 5396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.1
I20260928 12:21:32.051553 5396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:39533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:40853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.1
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:32.051775 5396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:32.051993 5396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:32.057933 5402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:32.058048 5404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:32.058214 5396 server_base.cc:1034] running on GCE node
W20260928 12:21:32.057940 5401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:32.058477 5396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:32.058755 5396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:32.059900 5396 hybrid_clock.cc:648] HybridClock initialized: now 1790598092059880 us; error 36 us; skew 500 ppm
Sep 28 12:21:32 dist-test-slave-920p krb5kdc[5289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598092, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:32.062625 5396 init.cc:377] Logged in from keytab as kudu/127.2.56.1@KRBTEST.COM (short username kudu)
I20260928 12:21:32.063087 5396 webserver.cc:460] Webserver started at http://127.2.56.1:39975/ using document root <none> and password file <none>
I20260928 12:21:32.063306 5396 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:32.063354 5396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:32.063488 5396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:32.064666 5396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-0/data/instance:
uuid: "f3c3a71518e44098bfd7d1a054067ab1"
format_stamp: "Formatted at 2026-09-28 12:21:32 on dist-test-slave-920p"
server_key: "f27b88bee009f2b4be69d286bdadeec7"
server_key_iv: "16ebf936b6002a00d260fb5e702466ee"
server_key_version: "encryptionkey@0"
I20260928 12:21:32.065029 5396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance:
uuid: "f3c3a71518e44098bfd7d1a054067ab1"
format_stamp: "Formatted at 2026-09-28 12:21:32 on dist-test-slave-920p"
server_key: "f27b88bee009f2b4be69d286bdadeec7"
server_key_iv: "16ebf936b6002a00d260fb5e702466ee"
server_key_version: "encryptionkey@0"
I20260928 12:21:32.066409 5396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:21:32.067092 5411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:32.067240 5396 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:32.067323 5396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-0/wal
uuid: "f3c3a71518e44098bfd7d1a054067ab1"
format_stamp: "Formatted at 2026-09-28 12:21:32 on dist-test-slave-920p"
server_key: "f27b88bee009f2b4be69d286bdadeec7"
server_key_iv: "16ebf936b6002a00d260fb5e702466ee"
server_key_version: "encryptionkey@0"
I20260928 12:21:32.067404 5396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:32.090173 5396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:32.092536 5396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:32.092660 5396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:32.092881 5396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:21:32.093174 5396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:32.093204 5396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:32.093230 5396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:32.093246 5396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:32.099776 5396 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.1:42067
I20260928 12:21:32.099828 5524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.1:42067 every 8 connection(s)
I20260928 12:21:32.100198 5396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
I20260928 12:21:32.106078 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 5396
I20260928 12:21:32.106184 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance
I20260928 12:21:32.106431 2272 external_mini_cluster.cc:1428] Setting key d851a294ca23d89e9443f8ac9787c4ed
Sep 28 12:21:32 dist-test-slave-920p krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598092, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:32.110695 5373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:32.100993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:44515 (local address 127.2.56.62:40853)
0928 12:21:32.101065 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:32.101069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:32.101306 (+ 237us) server_negotiation.cc:407] Connection header received
0928 12:21:32.101911 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:32.101912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:32.101923 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:32.101996 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:32.103057 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:32.103640 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:32.104704 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:32.104955 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:32.105029 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:32.107833 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:32.107835 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:32.107835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:32.107844 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:32.109357 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:32.109740 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:32.109741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:32.109741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:32.109803 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:32.110032 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:32.110033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:32.110034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:32.110179 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:32.110245 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:32.110488 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 12:21:32.110596 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 12:21:32.110807 5527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:32.100979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:40853 (local address 127.2.56.1:44515)
0928 12:21:32.101169 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:32.101211 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:32.101815 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:32.102082 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:32.102086 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:32.102726 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:32.102968 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:32.102973 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:32.103710 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:32.103711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:32.104594 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:32.104596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:32.105045 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:32.105046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:32.105169 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:32.105901 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:32.105912 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:32.107740 (+ 1828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:32.109433 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:32.109440 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:32.109442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:32.109671 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:32.109876 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:32.109876 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:32.109877 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:32.109978 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:32.110232 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:32.110236 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:32.110373 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 12:21:32.110460 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 12:21:32.110599 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":101,"threads_started":1}
I20260928 12:21:32.111330 5525 heartbeater.cc:347] Connected to a master server at 127.2.56.62:40853
I20260928 12:21:32.111459 5525 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:32.111686 5525 heartbeater.cc:511] Master 127.2.56.62:40853 requested a full tablet report, sending...
I20260928 12:21:32.112291 5334 ts_manager.cc:194] Registered new tserver with Master: f3c3a71518e44098bfd7d1a054067ab1 (127.2.56.1:42067)
I20260928 12:21:32.112958 5334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.1@KRBTEST.COM'} at 127.2.56.1:44515
WARNING: no policy specified for kudu/127.2.56.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab.
Entry for principal kudu/127.2.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab.
I20260928 12:21:32.132547 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.2:0
--local_ip_for_outbound_sockets=127.2.56.2
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:40853
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:39533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:32.220532 5530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:32.220805 5530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:32.220834 5530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:32.222261 5530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:32.222312 5530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:32.222345 5530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.2
I20260928 12:21:32.224224 5530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:39533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:40853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.2
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:32.224534 5530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:32.224784 5530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:32.230237 5538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:32.230237 5536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:32.230291 5530 server_base.cc:1034] running on GCE node
W20260928 12:21:32.230242 5535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:32.230651 5530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:32.230917 5530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:32.232072 5530 hybrid_clock.cc:648] HybridClock initialized: now 1790598092232057 us; error 34 us; skew 500 ppm
Sep 28 12:21:32 dist-test-slave-920p krb5kdc[5289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598092, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:32.234750 5530 init.cc:377] Logged in from keytab as kudu/127.2.56.2@KRBTEST.COM (short username kudu)
I20260928 12:21:32.235253 5530 webserver.cc:460] Webserver started at http://127.2.56.2:44149/ using document root <none> and password file <none>
I20260928 12:21:32.235482 5530 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:32.235533 5530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:32.235668 5530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:32.236832 5530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-1/data/instance:
uuid: "0b9664bec4ce4f9287fb62ef262214db"
format_stamp: "Formatted at 2026-09-28 12:21:32 on dist-test-slave-920p"
server_key: "963669dcb0fa193a8baa589ac67cba4b"
server_key_iv: "f89b70a82a54487336ea6540fab4b2d4"
server_key_version: "encryptionkey@0"
I20260928 12:21:32.237195 5530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance:
uuid: "0b9664bec4ce4f9287fb62ef262214db"
format_stamp: "Formatted at 2026-09-28 12:21:32 on dist-test-slave-920p"
server_key: "963669dcb0fa193a8baa589ac67cba4b"
server_key_iv: "f89b70a82a54487336ea6540fab4b2d4"
server_key_version: "encryptionkey@0"
I20260928 12:21:32.238523 5530 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:21:32.239192 5545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:32.239357 5530 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:32.239450 5530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-1/wal
uuid: "0b9664bec4ce4f9287fb62ef262214db"
format_stamp: "Formatted at 2026-09-28 12:21:32 on dist-test-slave-920p"
server_key: "963669dcb0fa193a8baa589ac67cba4b"
server_key_iv: "f89b70a82a54487336ea6540fab4b2d4"
server_key_version: "encryptionkey@0"
I20260928 12:21:32.239512 5530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:32.249168 5530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:32.250972 5530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:32.251112 5530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:32.251324 5530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:21:32.251658 5530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:32.251698 5530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:32.251726 5530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:32.251750 5530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:32.258105 5530 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.2:34583
I20260928 12:21:32.258157 5658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.2:34583 every 8 connection(s)
I20260928 12:21:32.258491 5530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
Sep 28 12:21:32 dist-test-slave-920p krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598092, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:32.267352 5373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:32.259192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:54267 (local address 127.2.56.62:40853)
0928 12:21:32.259242 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:32.259244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:32.259447 (+ 203us) server_negotiation.cc:407] Connection header received
0928 12:21:32.260028 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:32.260030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:32.260043 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:32.260103 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:32.261100 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:32.261705 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:32.262747 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:32.262961 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:32.263018 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:32.265093 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:32.265095 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:32.265096 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:32.265103 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:32.266256 (+ 1153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:32.266622 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:32.266623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:32.266623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:32.266667 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:32.266841 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:32.266841 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:32.266842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:32.266948 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:32.266994 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:32.267212 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 12:21:32.267277 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:21:32.267575 5661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:32.259167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:40853 (local address 127.2.56.2:54267)
0928 12:21:32.259310 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:32.259351 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:32.259918 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:32.260181 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:32.260186 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:32.260782 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:32.261007 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:32.261012 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:32.261796 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:32.261797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:32.262668 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:32.262671 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:32.263028 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:32.263029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:32.263159 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:32.263618 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:32.263627 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:32.265010 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:32.266322 (+ 1312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:32.266328 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:32.266329 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:32.266568 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:32.266708 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:32.266709 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:32.266709 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:32.266801 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:32.266989 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:32.266992 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:32.267117 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 12:21:32.267198 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 12:21:32.267324 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":70,"threads_started":1}
I20260928 12:21:32.267724 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 5530
I20260928 12:21:32.267819 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance
I20260928 12:21:32.267999 2272 external_mini_cluster.cc:1428] Setting key bc1c43f69ad03310a18072b0ec569061
I20260928 12:21:32.268086 5659 heartbeater.cc:347] Connected to a master server at 127.2.56.62:40853
I20260928 12:21:32.268182 5659 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:32.268368 5659 heartbeater.cc:511] Master 127.2.56.62:40853 requested a full tablet report, sending...
I20260928 12:21:32.268728 5334 ts_manager.cc:194] Registered new tserver with Master: 0b9664bec4ce4f9287fb62ef262214db (127.2.56.2:34583)
I20260928 12:21:32.269102 5334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.2@KRBTEST.COM'} at 127.2.56.2:54267
WARNING: no policy specified for kudu/127.2.56.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab.
Entry for principal kudu/127.2.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab.
I20260928 12:21:32.291862 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.3:0
--local_ip_for_outbound_sockets=127.2.56.3
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:40853
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:39533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:32.378851 5664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:32.379114 5664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:32.379143 5664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:32.380455 5664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:32.380501 5664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:32.380527 5664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.3
I20260928 12:21:32.382105 5664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:39533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:40853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.3
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:32.382359 5664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:32.382606 5664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:32.387764 5670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:32.387775 5672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:32.387780 5669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:32.388003 5664 server_base.cc:1034] running on GCE node
I20260928 12:21:32.388231 5664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:32.388504 5664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:32.389632 5664 hybrid_clock.cc:648] HybridClock initialized: now 1790598092389619 us; error 32 us; skew 500 ppm
Sep 28 12:21:32 dist-test-slave-920p krb5kdc[5289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598092, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:32.392316 5664 init.cc:377] Logged in from keytab as kudu/127.2.56.3@KRBTEST.COM (short username kudu)
I20260928 12:21:32.392795 5664 webserver.cc:460] Webserver started at http://127.2.56.3:38597/ using document root <none> and password file <none>
I20260928 12:21:32.393033 5664 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:32.393081 5664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:32.393214 5664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:32.394378 5664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-2/data/instance:
uuid: "a09e0c353bf44f72ae4aba50d0aea164"
format_stamp: "Formatted at 2026-09-28 12:21:32 on dist-test-slave-920p"
server_key: "a914069ea9181045dc4596bc02414bc9"
server_key_iv: "cce70add61c37c76172d529d0a38fff5"
server_key_version: "encryptionkey@0"
I20260928 12:21:32.394740 5664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance:
uuid: "a09e0c353bf44f72ae4aba50d0aea164"
format_stamp: "Formatted at 2026-09-28 12:21:32 on dist-test-slave-920p"
server_key: "a914069ea9181045dc4596bc02414bc9"
server_key_iv: "cce70add61c37c76172d529d0a38fff5"
server_key_version: "encryptionkey@0"
I20260928 12:21:32.396045 5664 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:21:32.396754 5679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:32.396904 5664 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:32.396982 5664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-2/wal
uuid: "a09e0c353bf44f72ae4aba50d0aea164"
format_stamp: "Formatted at 2026-09-28 12:21:32 on dist-test-slave-920p"
server_key: "a914069ea9181045dc4596bc02414bc9"
server_key_iv: "cce70add61c37c76172d529d0a38fff5"
server_key_version: "encryptionkey@0"
I20260928 12:21:32.397051 5664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:32.413349 5664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:32.415730 5664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:32.415879 5664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:32.416136 5664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:21:32.416471 5664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:32.416505 5664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:32.416538 5664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:32.416553 5664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:32.422858 5664 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.3:42303
I20260928 12:21:32.422916 5792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.3:42303 every 8 connection(s)
I20260928 12:21:32.423261 5664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
I20260928 12:21:32.426838 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 5664
I20260928 12:21:32.426915 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance
I20260928 12:21:32.427119 2272 external_mini_cluster.cc:1428] Setting key 833e2cb483323a6ff66fbc96286b61e3
Sep 28 12:21:32 dist-test-slave-920p krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598092, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:32.432826 5373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:32.423987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:36177 (local address 127.2.56.62:40853)
0928 12:21:32.424047 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:32.424049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:32.424245 (+ 196us) server_negotiation.cc:407] Connection header received
0928 12:21:32.424870 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:32.424871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:32.424885 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:32.424947 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:32.426038 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:32.426612 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:32.427989 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:32.428220 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:32.428278 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:32.430571 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:32.430572 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:32.430573 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:32.430580 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:32.431717 (+ 1137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:32.432076 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:32.432076 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:32.432077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:32.432133 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:32.432309 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:32.432310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:32.432310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:32.432415 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:32.432467 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:32.432690 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 12:21:32.432751 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 12:21:32.433063 5795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:32.423961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:40853 (local address 127.2.56.3:36177)
0928 12:21:32.424108 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:32.424150 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:32.424736 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:32.425033 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:32.425038 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:32.425699 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:32.425936 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:32.425942 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:32.426712 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:32.426713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:32.427897 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:32.427900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:32.428309 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:32.428310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:32.428436 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:32.428925 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:32.428934 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:32.430494 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:32.431782 (+ 1288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:32.431788 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:32.431791 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:32.432019 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:32.432176 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:32.432177 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:32.432178 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:32.432270 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:32.432459 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:32.432462 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:32.432591 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 12:21:32.432675 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 12:21:32.432807 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":69,"threads_started":1}
I20260928 12:21:32.433531 5793 heartbeater.cc:347] Connected to a master server at 127.2.56.62:40853
I20260928 12:21:32.433626 5793 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:32.433794 5793 heartbeater.cc:511] Master 127.2.56.62:40853 requested a full tablet report, sending...
I20260928 12:21:32.434161 5334 ts_manager.cc:194] Registered new tserver with Master: a09e0c353bf44f72ae4aba50d0aea164 (127.2.56.3:42303)
I20260928 12:21:32.434523 5334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.3@KRBTEST.COM'} at 127.2.56.3:36177
I20260928 12:21:32.438625 2272 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 12:21:32 dist-test-slave-920p krb5kdc[5289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598092, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 12:21:32 dist-test-slave-920p krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598092, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:32.460947 5373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:32.454240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56826 (local address 127.2.56.62:40853)
0928 12:21:32.454315 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:32.454318 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:32.454418 (+ 100us) server_negotiation.cc:407] Connection header received
0928 12:21:32.454499 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:32.454500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:32.454514 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:32.454564 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:32.455139 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:32.455675 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:32.456419 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:32.456618 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:32.456672 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:32.458809 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:32.458810 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:32.458811 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:32.458819 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:32.459899 (+ 1080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:32.460181 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:32.460182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:32.460183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:32.460234 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:32.460394 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:32.460395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:32.460396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:32.460559 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:32.460655 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:32.460744 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 12:21:32.460833 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 12:21:32.461167 5334 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56826
I20260928 12:21:32.461737 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 5396
I20260928 12:21:32.466423 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 5530
I20260928 12:21:32.470548 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 5664
I20260928 12:21:32.474702 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 5302
2026-09-28T12:21:32Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2091 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:21:32 dist-test-slave-920p krb5kdc[5805](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:21:32 dist-test-slave-920p krb5kdc[5805](info): set up 2 sockets
Sep 28 12:21:32 dist-test-slave-920p krb5kdc[5805](info): commencing operation
krb5kdc: starting...
W20260928 12:21:33.429407 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:21:33 dist-test-slave-920p krb5kdc[5805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:21:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:21:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
I20260928 12:21:33.555966 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:33141
--webserver_interface=127.2.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:33141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:33.646066 5818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:33.646345 5818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:33.646379 5818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:33.648015 5818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:21:33.648080 5818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:33.648102 5818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:33.648123 5818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:21:33.648140 5818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:21:33.649901 5818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:33141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:33141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:33.650213 5818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:33.650470 5818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:33.655937 5824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:33.655941 5826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:33.655936 5823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:33.656189 5818 server_base.cc:1034] running on GCE node
I20260928 12:21:33.656474 5818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:33.656841 5818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:33.658020 5818 hybrid_clock.cc:648] HybridClock initialized: now 1790598093657999 us; error 40 us; skew 500 ppm
Sep 28 12:21:33 dist-test-slave-920p krb5kdc[5805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598093, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:33.661293 5818 init.cc:377] Logged in from keytab as kudu/127.2.56.62@KRBTEST.COM (short username kudu)
I20260928 12:21:33.661847 5818 webserver.cc:460] Webserver started at http://127.2.56.62:41401/ using document root <none> and password file <none>
I20260928 12:21:33.662092 5818 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:33.662139 5818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:33.662256 5818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:33.663496 5818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "43670041b5944a41b5564b8dcf60ae69"
format_stamp: "Formatted at 2026-09-28 12:21:33 on dist-test-slave-920p"
server_key: "00f7539ba67a8fa018b58fb7da944d28"
server_key_iv: "2165ef3d3bb410cd22c25b3258fbecb1"
server_key_version: "encryptionkey@0"
I20260928 12:21:33.663861 5818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "43670041b5944a41b5564b8dcf60ae69"
format_stamp: "Formatted at 2026-09-28 12:21:33 on dist-test-slave-920p"
server_key: "00f7539ba67a8fa018b58fb7da944d28"
server_key_iv: "2165ef3d3bb410cd22c25b3258fbecb1"
server_key_version: "encryptionkey@0"
I20260928 12:21:33.665405 5818 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:21:33.666291 5833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:33.666487 5818 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:21:33.666577 5818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "43670041b5944a41b5564b8dcf60ae69"
format_stamp: "Formatted at 2026-09-28 12:21:33 on dist-test-slave-920p"
server_key: "00f7539ba67a8fa018b58fb7da944d28"
server_key_iv: "2165ef3d3bb410cd22c25b3258fbecb1"
server_key_version: "encryptionkey@0"
I20260928 12:21:33.666662 5818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:33.683940 5818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:33.686719 5818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:33.686900 5818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:33.691747 5818 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:33141
I20260928 12:21:33.691784 5885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:33141 every 8 connection(s)
I20260928 12:21:33.692401 5818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:21:33.693149 5886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:33.695763 5886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43670041b5944a41b5564b8dcf60ae69: Bootstrap starting.
I20260928 12:21:33.696379 5886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43670041b5944a41b5564b8dcf60ae69: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:33.696650 5886 log.cc:824] T 00000000000000000000000000000000 P 43670041b5944a41b5564b8dcf60ae69: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:33.697353 5886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43670041b5944a41b5564b8dcf60ae69: No bootstrap required, opened a new log
I20260928 12:21:33.699018 5886 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43670041b5944a41b5564b8dcf60ae69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43670041b5944a41b5564b8dcf60ae69" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 33141 } }
I20260928 12:21:33.699172 5886 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43670041b5944a41b5564b8dcf60ae69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:33.699194 5886 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43670041b5944a41b5564b8dcf60ae69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43670041b5944a41b5564b8dcf60ae69, State: Initialized, Role: FOLLOWER
I20260928 12:21:33.699317 5886 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43670041b5944a41b5564b8dcf60ae69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43670041b5944a41b5564b8dcf60ae69" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 33141 } }
I20260928 12:21:33.699393 5886 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43670041b5944a41b5564b8dcf60ae69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:21:33.699426 5886 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43670041b5944a41b5564b8dcf60ae69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:21:33.699463 5886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43670041b5944a41b5564b8dcf60ae69 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:33.700320 5886 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43670041b5944a41b5564b8dcf60ae69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43670041b5944a41b5564b8dcf60ae69" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 33141 } }
I20260928 12:21:33.700415 5886 leader_election.cc:302] T 00000000000000000000000000000000 P 43670041b5944a41b5564b8dcf60ae69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43670041b5944a41b5564b8dcf60ae69; no voters:
I20260928 12:21:33.700546 5886 leader_election.cc:288] T 00000000000000000000000000000000 P 43670041b5944a41b5564b8dcf60ae69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:21:33.700601 5889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43670041b5944a41b5564b8dcf60ae69 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:21:33.700779 5889 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43670041b5944a41b5564b8dcf60ae69 [term 1 LEADER]: Becoming Leader. State: Replica: 43670041b5944a41b5564b8dcf60ae69, State: Running, Role: LEADER
I20260928 12:21:33.700798 5886 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43670041b5944a41b5564b8dcf60ae69 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:21:33.700876 5889 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43670041b5944a41b5564b8dcf60ae69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43670041b5944a41b5564b8dcf60ae69" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 33141 } }
I20260928 12:21:33.701279 5890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43670041b5944a41b5564b8dcf60ae69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43670041b5944a41b5564b8dcf60ae69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43670041b5944a41b5564b8dcf60ae69" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 33141 } } }
I20260928 12:21:33.701316 5891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43670041b5944a41b5564b8dcf60ae69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43670041b5944a41b5564b8dcf60ae69. Latest consensus state: current_term: 1 leader_uuid: "43670041b5944a41b5564b8dcf60ae69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43670041b5944a41b5564b8dcf60ae69" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 33141 } } }
I20260928 12:21:33.701510 5891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43670041b5944a41b5564b8dcf60ae69 [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:33.701499 5890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43670041b5944a41b5564b8dcf60ae69 [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:33.701707 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 5818
I20260928 12:21:33.701819 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:21:33.701895 5898 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:21:33.702041 2272 external_mini_cluster.cc:1428] Setting key 2add79b18c50a58a329fa59df0be6702
I20260928 12:21:33.702459 5898 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:21:33.704345 5898 catalog_manager.cc:1347] Generated new cluster ID: 215fa1406b814af080d99a0fb71b4843
I20260928 12:21:33.704396 5898 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:21:33 dist-test-slave-920p krb5kdc[5805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:33.715085 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:33.702753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56140 (local address 127.2.56.62:33141)
0928 12:21:33.702923 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:33.702935 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:33.703808 (+ 873us) server_negotiation.cc:407] Connection header received
0928 12:21:33.703954 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:33.703962 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:33.704012 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:33.704141 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:33.704927 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:33.706497 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:33.707565 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:33.708270 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:33.708393 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:33.711292 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:33.711297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:33.711297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:33.711310 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:33.713343 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:33.713762 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:33.713766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:33.713766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:33.713878 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:33.714056 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:33.714057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:33.714058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:33.714405 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:33.714571 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:33.714665 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 12:21:33.714815 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
I20260928 12:21:33.717483 5898 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:21:33.718045 5898 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:21:33.726450 5898 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43670041b5944a41b5564b8dcf60ae69: Generated new TSK 0
I20260928 12:21:33.726673 5898 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab.
Entry for principal kudu/127.2.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab.
I20260928 12:21:33.756069 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.1:0
--local_ip_for_outbound_sockets=127.2.56.1
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:33141
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:33.870728 5912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:33.871011 5912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:33.871037 5912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:33.872262 5912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:33.872310 5912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:33.872337 5912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.1
I20260928 12:21:33.873962 5912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:33141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.1
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:33.874230 5912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:33.874464 5912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:33.880249 5918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:33.880259 5917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:33.880266 5920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:33.880313 5912 server_base.cc:1034] running on GCE node
I20260928 12:21:33.880690 5912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:33.880993 5912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:33.882140 5912 hybrid_clock.cc:648] HybridClock initialized: now 1790598093882121 us; error 35 us; skew 500 ppm
Sep 28 12:21:33 dist-test-slave-920p krb5kdc[5805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598093, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:33.884953 5912 init.cc:377] Logged in from keytab as kudu/127.2.56.1@KRBTEST.COM (short username kudu)
I20260928 12:21:33.885461 5912 webserver.cc:460] Webserver started at http://127.2.56.1:40081/ using document root <none> and password file <none>
I20260928 12:21:33.885681 5912 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:33.885723 5912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:33.885828 5912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:33.887059 5912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-0/data/instance:
uuid: "e721964c76254c11aa71b507e1553c4d"
format_stamp: "Formatted at 2026-09-28 12:21:33 on dist-test-slave-920p"
server_key: "3a9990481fd5c734db82e0588d4dd3bc"
server_key_iv: "65e3eaa63863037eb448665bf2381d3b"
server_key_version: "encryptionkey@0"
I20260928 12:21:33.887395 5912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance:
uuid: "e721964c76254c11aa71b507e1553c4d"
format_stamp: "Formatted at 2026-09-28 12:21:33 on dist-test-slave-920p"
server_key: "3a9990481fd5c734db82e0588d4dd3bc"
server_key_iv: "65e3eaa63863037eb448665bf2381d3b"
server_key_version: "encryptionkey@0"
I20260928 12:21:33.888691 5912 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:21:33.889430 5927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:33.889581 5912 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:33.889673 5912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-0/wal
uuid: "e721964c76254c11aa71b507e1553c4d"
format_stamp: "Formatted at 2026-09-28 12:21:33 on dist-test-slave-920p"
server_key: "3a9990481fd5c734db82e0588d4dd3bc"
server_key_iv: "65e3eaa63863037eb448665bf2381d3b"
server_key_version: "encryptionkey@0"
I20260928 12:21:33.889734 5912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:33.927613 5912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:33.929864 5912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:33.930008 5912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:33.930251 5912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:21:33.930574 5912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:33.930608 5912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:33.930632 5912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:33.930648 5912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:33.937194 5912 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.1:37039
I20260928 12:21:33.937299 6040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.1:37039 every 8 connection(s)
I20260928 12:21:33.937670 5912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
I20260928 12:21:33.943158 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 5912
I20260928 12:21:33.943269 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance
I20260928 12:21:33.943502 2272 external_mini_cluster.cc:1428] Setting key 10b3ba6235ffed1ef1a8ca72a767f996
Sep 28 12:21:33 dist-test-slave-920p krb5kdc[5805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598093, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:33.948119 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:33.938503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:55899 (local address 127.2.56.62:33141)
0928 12:21:33.938571 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:33.938573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:33.938788 (+ 215us) server_negotiation.cc:407] Connection header received
0928 12:21:33.939905 (+ 1117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:33.939906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:33.939927 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:33.939983 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:33.940734 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:33.941323 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:33.942442 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:33.942678 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:33.942738 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:33.945153 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:33.945155 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:33.945155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:33.945172 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:33.946650 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:33.947091 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:33.947092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:33.947093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:33.947159 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:33.947405 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:33.947406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:33.947406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:33.947545 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:33.947605 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:33.947906 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 12:21:33.948013 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 12:21:33.948343 6043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:33.938469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:33141 (local address 127.2.56.1:55899)
0928 12:21:33.938653 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:33.938695 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:33.939775 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:33.940073 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:33.940079 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:33.940388 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:33.940627 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:33.940632 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:33.941413 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:33.941414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:33.942344 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:33.942346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:33.942756 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:33.942757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:33.942901 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:33.943523 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:33.943536 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:33.945060 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:33.946732 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:33.946742 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:33.946745 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:33.947018 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:33.947213 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:33.947214 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:33.947215 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:33.947349 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:33.947599 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:33.947608 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:33.947779 (+ 171us) client_negotiation.cc:769] Sending connection context
0928 12:21:33.947880 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 12:21:33.948079 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":95,"threads_started":1}
I20260928 12:21:33.948858 6041 heartbeater.cc:347] Connected to a master server at 127.2.56.62:33141
I20260928 12:21:33.949002 6041 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:33.949255 6041 heartbeater.cc:511] Master 127.2.56.62:33141 requested a full tablet report, sending...
I20260928 12:21:33.949970 5850 ts_manager.cc:194] Registered new tserver with Master: e721964c76254c11aa71b507e1553c4d (127.2.56.1:37039)
I20260928 12:21:33.950639 5850 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.1@KRBTEST.COM'} at 127.2.56.1:55899
WARNING: no policy specified for kudu/127.2.56.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab.
Entry for principal kudu/127.2.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab.
I20260928 12:21:33.972604 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.2:0
--local_ip_for_outbound_sockets=127.2.56.2
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:33141
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:34.060560 6046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:34.060791 6046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:34.060822 6046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:34.062052 6046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:34.062103 6046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:34.062127 6046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.2
I20260928 12:21:34.063686 6046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:33141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.2
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:34.063915 6046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:34.064167 6046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:34.069293 6051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:34.069305 6046 server_base.cc:1034] running on GCE node
W20260928 12:21:34.069367 6052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:34.069420 6054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:34.069716 6046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:34.069972 6046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:34.071127 6046 hybrid_clock.cc:648] HybridClock initialized: now 1790598094071119 us; error 34 us; skew 500 ppm
Sep 28 12:21:34 dist-test-slave-920p krb5kdc[5805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598094, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:34.073766 6046 init.cc:377] Logged in from keytab as kudu/127.2.56.2@KRBTEST.COM (short username kudu)
I20260928 12:21:34.074234 6046 webserver.cc:460] Webserver started at http://127.2.56.2:42731/ using document root <none> and password file <none>
I20260928 12:21:34.074468 6046 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:34.074519 6046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:34.074642 6046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:34.075738 6046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-1/data/instance:
uuid: "47ab74e40b0b4951b6b2b7f5f4644a20"
format_stamp: "Formatted at 2026-09-28 12:21:34 on dist-test-slave-920p"
server_key: "f2b57e14e7395bd35b6df541af868afc"
server_key_iv: "ca347214fd95baec96880ad357317c2d"
server_key_version: "encryptionkey@0"
I20260928 12:21:34.076108 6046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance:
uuid: "47ab74e40b0b4951b6b2b7f5f4644a20"
format_stamp: "Formatted at 2026-09-28 12:21:34 on dist-test-slave-920p"
server_key: "f2b57e14e7395bd35b6df541af868afc"
server_key_iv: "ca347214fd95baec96880ad357317c2d"
server_key_version: "encryptionkey@0"
I20260928 12:21:34.077494 6046 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 12:21:34.078326 6061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:34.078531 6046 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 12:21:34.078621 6046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-1/wal
uuid: "47ab74e40b0b4951b6b2b7f5f4644a20"
format_stamp: "Formatted at 2026-09-28 12:21:34 on dist-test-slave-920p"
server_key: "f2b57e14e7395bd35b6df541af868afc"
server_key_iv: "ca347214fd95baec96880ad357317c2d"
server_key_version: "encryptionkey@0"
I20260928 12:21:34.078687 6046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:34.109928 6046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:34.112213 6046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:34.112406 6046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:34.112709 6046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:21:34.113125 6046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:34.113161 6046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:34.113200 6046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:34.113229 6046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:34.120420 6046 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.2:38703
I20260928 12:21:34.120460 6174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.2:38703 every 8 connection(s)
I20260928 12:21:34.120870 6046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
Sep 28 12:21:34 dist-test-slave-920p krb5kdc[5805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598094, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:34.128481 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 6046
I20260928 12:21:34.128594 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance
I20260928 12:21:34.128803 2272 external_mini_cluster.cc:1428] Setting key d89f543ecd1371f97147df6b85aca0d6
I20260928 12:21:34.132112 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:34.121787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:33603 (local address 127.2.56.62:33141)
0928 12:21:34.121870 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:34.121872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:34.122162 (+ 290us) server_negotiation.cc:407] Connection header received
0928 12:21:34.123295 (+ 1133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:34.123297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:34.123311 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:34.123379 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:34.124174 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:34.124759 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:34.126041 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:34.126264 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:34.126327 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:34.129044 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:34.129047 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:34.129047 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:34.129058 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:34.130586 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:34.131092 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:34.131092 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:34.131093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:34.131144 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:34.131397 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:34.131399 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:34.131399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:34.131534 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:34.131594 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:34.131894 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 12:21:34.132011 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 12:21:34.132326 6177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:34.121759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:33141 (local address 127.2.56.2:33603)
0928 12:21:34.122003 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:34.122052 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:34.123184 (+ 1132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:34.123461 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:34.123466 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:34.123811 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:34.124078 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:34.124085 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:34.124866 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:34.124868 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:34.125939 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:34.125944 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:34.126345 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:34.126346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:34.126514 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:34.127163 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:34.127174 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:34.128916 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:34.130669 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:34.130678 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:34.130681 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:34.131023 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:34.131210 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:34.131211 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:34.131211 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:34.131337 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:34.131588 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:34.131595 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:34.131767 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 12:21:34.131866 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 12:21:34.132058 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":105,"threads_started":1}
I20260928 12:21:34.132830 6175 heartbeater.cc:347] Connected to a master server at 127.2.56.62:33141
I20260928 12:21:34.132952 6175 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:34.133157 6175 heartbeater.cc:511] Master 127.2.56.62:33141 requested a full tablet report, sending...
I20260928 12:21:34.133584 5850 ts_manager.cc:194] Registered new tserver with Master: 47ab74e40b0b4951b6b2b7f5f4644a20 (127.2.56.2:38703)
I20260928 12:21:34.134016 5850 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.2@KRBTEST.COM'} at 127.2.56.2:33603
WARNING: no policy specified for kudu/127.2.56.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab.
Entry for principal kudu/127.2.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab.
I20260928 12:21:34.156291 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.3:0
--local_ip_for_outbound_sockets=127.2.56.3
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:33141
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:34.244489 6180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:34.244740 6180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:34.244773 6180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:34.246021 6180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:34.246068 6180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:34.246104 6180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.3
I20260928 12:21:34.247720 6180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:33141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.3
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:34.248000 6180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:34.248234 6180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:34.253432 6188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:34.253444 6185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:34.253497 6186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:34.253754 6180 server_base.cc:1034] running on GCE node
I20260928 12:21:34.253981 6180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:34.254199 6180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:34.255329 6180 hybrid_clock.cc:648] HybridClock initialized: now 1790598094255316 us; error 30 us; skew 500 ppm
Sep 28 12:21:34 dist-test-slave-920p krb5kdc[5805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598094, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:34.257956 6180 init.cc:377] Logged in from keytab as kudu/127.2.56.3@KRBTEST.COM (short username kudu)
I20260928 12:21:34.258392 6180 webserver.cc:460] Webserver started at http://127.2.56.3:34927/ using document root <none> and password file <none>
I20260928 12:21:34.258591 6180 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:34.258632 6180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:34.258729 6180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:34.259816 6180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-2/data/instance:
uuid: "ffbbcf8d0962441b8e66a12ea4cf1ec8"
format_stamp: "Formatted at 2026-09-28 12:21:34 on dist-test-slave-920p"
server_key: "1ec6c088262a99d53b534098049c1445"
server_key_iv: "39cb563eeb54066829c2d349863278ff"
server_key_version: "encryptionkey@0"
I20260928 12:21:34.260133 6180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance:
uuid: "ffbbcf8d0962441b8e66a12ea4cf1ec8"
format_stamp: "Formatted at 2026-09-28 12:21:34 on dist-test-slave-920p"
server_key: "1ec6c088262a99d53b534098049c1445"
server_key_iv: "39cb563eeb54066829c2d349863278ff"
server_key_version: "encryptionkey@0"
I20260928 12:21:34.261365 6180 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 12:21:34.262048 6195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:34.262198 6180 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:34.262280 6180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-2/wal
uuid: "ffbbcf8d0962441b8e66a12ea4cf1ec8"
format_stamp: "Formatted at 2026-09-28 12:21:34 on dist-test-slave-920p"
server_key: "1ec6c088262a99d53b534098049c1445"
server_key_iv: "39cb563eeb54066829c2d349863278ff"
server_key_version: "encryptionkey@0"
I20260928 12:21:34.262341 6180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:34.270692 6180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:34.272748 6180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:34.272877 6180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:34.273087 6180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:21:34.273439 6180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:34.273483 6180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:34.273514 6180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:34.273536 6180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:34.280172 6180 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.3:36661
I20260928 12:21:34.280220 6308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.3:36661 every 8 connection(s)
I20260928 12:21:34.280580 6180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
I20260928 12:21:34.281474 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 6180
I20260928 12:21:34.281570 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance
I20260928 12:21:34.281780 2272 external_mini_cluster.cc:1428] Setting key 34eceaa20c00b3ff11796ab22eb63e6f
Sep 28 12:21:34 dist-test-slave-920p krb5kdc[5805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598094, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:34.290620 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:34.281377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:50695 (local address 127.2.56.62:33141)
0928 12:21:34.281447 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:34.281450 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:34.281710 (+ 260us) server_negotiation.cc:407] Connection header received
0928 12:21:34.283090 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:34.283091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:34.283102 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:34.283143 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:34.283805 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:34.284346 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:34.285454 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:34.285643 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:34.285689 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:34.287935 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:34.287937 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:34.287938 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:34.287945 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:34.289448 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:34.289805 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:34.289806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:34.289807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:34.289867 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:34.290058 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:34.290059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:34.290059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:34.290174 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:34.290222 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:34.290469 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 12:21:34.290539 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 12:21:34.290786 6311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:34.281365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:33141 (local address 127.2.56.3:50695)
0928 12:21:34.281557 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:34.281613 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:34.283011 (+ 1398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:34.283212 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:34.283217 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:34.283514 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:34.283742 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:34.283747 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:34.284428 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:34.284429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:34.285384 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:34.285387 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:34.285695 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:34.285695 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:34.285799 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:34.286307 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:34.286316 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:34.287841 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:34.289518 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:34.289524 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:34.289526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:34.289731 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:34.289927 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:34.289927 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:34.289928 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:34.290019 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:34.290215 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:34.290218 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:34.290359 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 12:21:34.290448 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 12:21:34.290579 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":93,"threads_started":1}
I20260928 12:21:34.291251 6309 heartbeater.cc:347] Connected to a master server at 127.2.56.62:33141
I20260928 12:21:34.291350 6309 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:34.291543 6309 heartbeater.cc:511] Master 127.2.56.62:33141 requested a full tablet report, sending...
I20260928 12:21:34.291908 5850 ts_manager.cc:194] Registered new tserver with Master: ffbbcf8d0962441b8e66a12ea4cf1ec8 (127.2.56.3:36661)
I20260928 12:21:34.292286 5850 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.3@KRBTEST.COM'} at 127.2.56.3:50695
I20260928 12:21:34.293493 2272 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 12:21:34 dist-test-slave-920p krb5kdc[5805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 12:21:34 dist-test-slave-920p krb5kdc[5805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.1@KRBTEST.COM
I20260928 12:21:34.316928 6316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:34.309296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45922 (local address 127.2.56.1:37039)
0928 12:21:34.309480 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:34.309486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:34.309496 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:21:34.309526 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:34.309528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:34.309544 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:34.309616 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:34.310235 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:34.311021 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:34.311762 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:34.312142 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:34.312204 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:34.314484 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:34.314494 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:34.314495 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:34.314502 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:34.315940 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:34.316250 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:34.316253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:34.316253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:34.316321 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:34.316465 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:34.316466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:34.316466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:34.316614 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:34.316729 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:34.316767 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 12:21:34.316833 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":85,"threads_started":1}
Sep 28 12:21:34 dist-test-slave-920p krb5kdc[5805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:34.324232 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:34.318312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56144 (local address 127.2.56.62:33141)
0928 12:21:34.318370 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:34.318372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:34.318379 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:21:34.318434 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:34.318435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:34.318445 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:34.318488 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:34.318885 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:34.319399 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:34.320028 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:34.320215 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:34.320265 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:34.322311 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:34.322313 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:34.322314 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:34.322320 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:34.323386 (+ 1066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:34.323653 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:34.323653 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:34.323654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:34.323695 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:34.323815 (+ 120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:34.323815 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:34.323816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:34.323962 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:34.324033 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:34.324101 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 12:21:34.324159 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 12:21:34.324496 5850 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56144
I20260928 12:21:34.324824 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 5912
I20260928 12:21:34.329967 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 6046
I20260928 12:21:34.334100 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 6180
I20260928 12:21:34.338085 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 5818
2026-09-28T12:21:34Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1863 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:21:34 dist-test-slave-920p krb5kdc[6321](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:21:34 dist-test-slave-920p krb5kdc[6321](info): set up 2 sockets
Sep 28 12:21:34 dist-test-slave-920p krb5kdc[6321](info): commencing operation
krb5kdc: starting...
W20260928 12:21:35.289526 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:21:35 dist-test-slave-920p krb5kdc[6321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:21:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:21:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
I20260928 12:21:35.403972 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:39411
--webserver_interface=127.2.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:43307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:39411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:35.489703 6334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:35.489948 6334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:35.489980 6334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:35.491179 6334 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:21:35.491217 6334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:35.491235 6334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:35.491252 6334 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:21:35.491266 6334 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:21:35.492758 6334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:43307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:39411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:39411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:35.493016 6334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:35.493233 6334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:35.498068 6340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:35.498091 6342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:35.498080 6339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:35.498509 6334 server_base.cc:1034] running on GCE node
I20260928 12:21:35.498747 6334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:35.499006 6334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:35.500152 6334 hybrid_clock.cc:648] HybridClock initialized: now 1790598095500143 us; error 30 us; skew 500 ppm
Sep 28 12:21:35 dist-test-slave-920p krb5kdc[6321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598095, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:35.502741 6334 init.cc:377] Logged in from keytab as kudu/127.2.56.62@KRBTEST.COM (short username kudu)
I20260928 12:21:35.503096 6334 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:35.503144 6334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:35.503365 6334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:35.504592 6334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "a6a3f307f8614f1ebeee387e3c1a34c9"
format_stamp: "Formatted at 2026-09-28 12:21:35 on dist-test-slave-920p"
server_key: "916ab01e96c192d2a647f176724ae899"
server_key_iv: "ac527c47977e712ac69af7e9a8e46bae"
server_key_version: "encryptionkey@0"
I20260928 12:21:35.504967 6334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "a6a3f307f8614f1ebeee387e3c1a34c9"
format_stamp: "Formatted at 2026-09-28 12:21:35 on dist-test-slave-920p"
server_key: "916ab01e96c192d2a647f176724ae899"
server_key_iv: "ac527c47977e712ac69af7e9a8e46bae"
server_key_version: "encryptionkey@0"
I20260928 12:21:35.506263 6334 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:21:35.506945 6348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:35.507082 6334 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:35.507156 6334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "a6a3f307f8614f1ebeee387e3c1a34c9"
format_stamp: "Formatted at 2026-09-28 12:21:35 on dist-test-slave-920p"
server_key: "916ab01e96c192d2a647f176724ae899"
server_key_iv: "ac527c47977e712ac69af7e9a8e46bae"
server_key_version: "encryptionkey@0"
I20260928 12:21:35.507227 6334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:35.517508 6334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:35.519253 6334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:35.519384 6334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:35.523523 6334 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:39411
I20260928 12:21:35.523546 6400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:39411 every 8 connection(s)
I20260928 12:21:35.523820 6334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:21:35.524734 6401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:35.527266 6401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6a3f307f8614f1ebeee387e3c1a34c9: Bootstrap starting.
I20260928 12:21:35.527840 6401 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6a3f307f8614f1ebeee387e3c1a34c9: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:35.528111 6401 log.cc:824] T 00000000000000000000000000000000 P a6a3f307f8614f1ebeee387e3c1a34c9: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:35.528453 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 6334
I20260928 12:21:35.528554 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:21:35.528723 2272 external_mini_cluster.cc:1428] Setting key bb409a34bcebb8f88c6ddb5c5860c2b3
I20260928 12:21:35.528790 6401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6a3f307f8614f1ebeee387e3c1a34c9: No bootstrap required, opened a new log
I20260928 12:21:35.531421 6401 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6a3f307f8614f1ebeee387e3c1a34c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a3f307f8614f1ebeee387e3c1a34c9" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39411 } }
I20260928 12:21:35.531603 6401 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6a3f307f8614f1ebeee387e3c1a34c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:35.531637 6401 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6a3f307f8614f1ebeee387e3c1a34c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a3f307f8614f1ebeee387e3c1a34c9, State: Initialized, Role: FOLLOWER
I20260928 12:21:35.531767 6401 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6a3f307f8614f1ebeee387e3c1a34c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a3f307f8614f1ebeee387e3c1a34c9" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39411 } }
I20260928 12:21:35.531836 6401 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6a3f307f8614f1ebeee387e3c1a34c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:21:35.531868 6401 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6a3f307f8614f1ebeee387e3c1a34c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:21:35.531912 6401 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6a3f307f8614f1ebeee387e3c1a34c9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:35.532577 6401 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6a3f307f8614f1ebeee387e3c1a34c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a3f307f8614f1ebeee387e3c1a34c9" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39411 } }
I20260928 12:21:35.532675 6401 leader_election.cc:302] T 00000000000000000000000000000000 P a6a3f307f8614f1ebeee387e3c1a34c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6a3f307f8614f1ebeee387e3c1a34c9; no voters:
I20260928 12:21:35.532826 6401 leader_election.cc:288] T 00000000000000000000000000000000 P a6a3f307f8614f1ebeee387e3c1a34c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:21:35.532855 6406 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6a3f307f8614f1ebeee387e3c1a34c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:21:35.532963 6406 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6a3f307f8614f1ebeee387e3c1a34c9 [term 1 LEADER]: Becoming Leader. State: Replica: a6a3f307f8614f1ebeee387e3c1a34c9, State: Running, Role: LEADER
I20260928 12:21:35.533044 6406 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6a3f307f8614f1ebeee387e3c1a34c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a3f307f8614f1ebeee387e3c1a34c9" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39411 } }
I20260928 12:21:35.533110 6401 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6a3f307f8614f1ebeee387e3c1a34c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:21:35.533406 6406 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6a3f307f8614f1ebeee387e3c1a34c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6a3f307f8614f1ebeee387e3c1a34c9. Latest consensus state: current_term: 1 leader_uuid: "a6a3f307f8614f1ebeee387e3c1a34c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a3f307f8614f1ebeee387e3c1a34c9" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39411 } } }
I20260928 12:21:35.533416 6407 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6a3f307f8614f1ebeee387e3c1a34c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6a3f307f8614f1ebeee387e3c1a34c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a3f307f8614f1ebeee387e3c1a34c9" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39411 } } }
I20260928 12:21:35.533578 6406 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6a3f307f8614f1ebeee387e3c1a34c9 [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:35.533591 6407 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6a3f307f8614f1ebeee387e3c1a34c9 [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:35.533835 6412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:21:35.534448 6412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:21:35.536266 6412 catalog_manager.cc:1347] Generated new cluster ID: f60c510e4e984c19aba4dc394dd4baf2
I20260928 12:21:35.536309 6412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:21:35 dist-test-slave-920p krb5kdc[6321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:35.541006 6405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:35.529245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59812 (local address 127.2.56.62:39411)
0928 12:21:35.529478 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:35.529491 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:35.529526 (+ 35us) server_negotiation.cc:407] Connection header received
0928 12:21:35.530160 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:35.530170 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:35.530237 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:35.530395 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:35.531044 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:35.532452 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:35.533617 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:35.534287 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:35.534379 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:35.537411 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:35.537416 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:35.537416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:35.537442 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:35.539360 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:35.539743 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:35.539748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:35.539748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:35.539858 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:35.540035 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:35.540036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:35.540036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:35.540356 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:35.540539 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:35.540622 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 12:21:35.540766 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":82,"threads_started":1}
I20260928 12:21:35.546355 6412 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:21:35.546871 6412 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:21:35.555984 6412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6a3f307f8614f1ebeee387e3c1a34c9: Generated new TSK 0
I20260928 12:21:35.556165 6412 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab.
Entry for principal kudu/127.2.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab.
I20260928 12:21:35.588682 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.1:0
--local_ip_for_outbound_sockets=127.2.56.1
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:39411
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:43307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:35.697640 6427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:35.697885 6427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:35.697909 6427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:35.699139 6427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:35.699184 6427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:35.699213 6427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.1
I20260928 12:21:35.700788 6427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:43307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:39411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.1
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:35.701042 6427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:35.701321 6427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:35.706516 6433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:35.706518 6432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:35.706542 6435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:35.706665 6427 server_base.cc:1034] running on GCE node
I20260928 12:21:35.707015 6427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:35.707263 6427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:35.708410 6427 hybrid_clock.cc:648] HybridClock initialized: now 1790598095708396 us; error 32 us; skew 500 ppm
Sep 28 12:21:35 dist-test-slave-920p krb5kdc[6321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598095, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:35.711262 6427 init.cc:377] Logged in from keytab as kudu/127.2.56.1@KRBTEST.COM (short username kudu)
I20260928 12:21:35.711598 6427 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:35.711643 6427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:35.711743 6427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:35.712896 6427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-0/data/instance:
uuid: "bd3ecd7b1284485989ff10cc2fe77b1f"
format_stamp: "Formatted at 2026-09-28 12:21:35 on dist-test-slave-920p"
server_key: "e17f2ee51522703afa65cd471ecc08b1"
server_key_iv: "af961239db01c412f2dfbd0aa1885344"
server_key_version: "encryptionkey@0"
I20260928 12:21:35.713209 6427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance:
uuid: "bd3ecd7b1284485989ff10cc2fe77b1f"
format_stamp: "Formatted at 2026-09-28 12:21:35 on dist-test-slave-920p"
server_key: "e17f2ee51522703afa65cd471ecc08b1"
server_key_iv: "af961239db01c412f2dfbd0aa1885344"
server_key_version: "encryptionkey@0"
I20260928 12:21:35.714488 6427 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:21:35.715137 6441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:35.715296 6427 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:21:35.715389 6427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-0/wal
uuid: "bd3ecd7b1284485989ff10cc2fe77b1f"
format_stamp: "Formatted at 2026-09-28 12:21:35 on dist-test-slave-920p"
server_key: "e17f2ee51522703afa65cd471ecc08b1"
server_key_iv: "af961239db01c412f2dfbd0aa1885344"
server_key_version: "encryptionkey@0"
I20260928 12:21:35.715448 6427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:35.735371 6427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:35.737852 6427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:35.737980 6427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:35.738198 6427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:21:35.738554 6427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:35.738595 6427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:35.738626 6427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:35.738642 6427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:35.746299 6427 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.1:34439
I20260928 12:21:35.746358 6554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.1:34439 every 8 connection(s)
I20260928 12:21:35.746657 6427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
Sep 28 12:21:35 dist-test-slave-920p krb5kdc[6321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598095, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:35.755486 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 6427
I20260928 12:21:35.755601 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance
I20260928 12:21:35.755836 2272 external_mini_cluster.cc:1428] Setting key cb5504cf3f085a10d04fe76d34e6229b
I20260928 12:21:35.757099 6405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:35.747503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:58657 (local address 127.2.56.62:39411)
0928 12:21:35.747576 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:35.747578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:35.747802 (+ 224us) server_negotiation.cc:407] Connection header received
0928 12:21:35.748385 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:35.748386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:35.748397 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:35.748455 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:35.749646 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:35.750231 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:35.751507 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:35.751737 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:35.751789 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:35.754090 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:35.754091 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:35.754092 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:35.754099 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:35.755400 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:35.756028 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:35.756030 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:35.756030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:35.756096 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:35.756351 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:35.756353 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:35.756354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:35.756489 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:35.756555 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:35.756890 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 12:21:35.757002 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 12:21:35.757321 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:35.747475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39411 (local address 127.2.56.1:58657)
0928 12:21:35.747659 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:35.747708 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:35.748297 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:35.748533 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:35.748537 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:35.748847 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:35.749552 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:35.749559 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:35.750341 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:35.750343 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:35.751406 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:35.751408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:35.751828 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:35.751829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:35.751968 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:35.752477 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:35.752485 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:35.753999 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:35.755498 (+ 1499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:35.755506 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:35.755508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:35.755945 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:35.756168 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:35.756169 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:35.756170 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:35.756283 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:35.756553 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:35.756560 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:35.756707 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 12:21:35.756855 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 12:21:35.757042 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":97,"threads_started":1}
I20260928 12:21:35.758029 6555 heartbeater.cc:347] Connected to a master server at 127.2.56.62:39411
I20260928 12:21:35.758152 6555 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:35.758404 6555 heartbeater.cc:511] Master 127.2.56.62:39411 requested a full tablet report, sending...
I20260928 12:21:35.759207 6365 ts_manager.cc:194] Registered new tserver with Master: bd3ecd7b1284485989ff10cc2fe77b1f (127.2.56.1:34439)
I20260928 12:21:35.760031 6365 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.1@KRBTEST.COM'} at 127.2.56.1:58657
WARNING: no policy specified for kudu/127.2.56.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab.
Entry for principal kudu/127.2.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab.
I20260928 12:21:35.785663 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.2:0
--local_ip_for_outbound_sockets=127.2.56.2
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:39411
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:43307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:35.874212 6560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:35.874471 6560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:35.874508 6560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:35.875751 6560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:35.875792 6560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:35.875828 6560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.2
I20260928 12:21:35.877432 6560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:43307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:39411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.2
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:35.877679 6560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:35.877904 6560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:35.883018 6566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:35.883004 6565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:35.883006 6568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:35.883251 6560 server_base.cc:1034] running on GCE node
I20260928 12:21:35.883487 6560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:35.883719 6560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:35.884863 6560 hybrid_clock.cc:648] HybridClock initialized: now 1790598095884849 us; error 34 us; skew 500 ppm
Sep 28 12:21:35 dist-test-slave-920p krb5kdc[6321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598095, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:35.888206 6560 init.cc:377] Logged in from keytab as kudu/127.2.56.2@KRBTEST.COM (short username kudu)
I20260928 12:21:35.888623 6560 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:35.888682 6560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:35.888829 6560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:35.890168 6560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-1/data/instance:
uuid: "93af5027e9c74cf6ae5aeb5a96e71290"
format_stamp: "Formatted at 2026-09-28 12:21:35 on dist-test-slave-920p"
server_key: "1f167ef43035d23eadb715ec982d42cd"
server_key_iv: "4908528f5f17875e047f457323517d37"
server_key_version: "encryptionkey@0"
I20260928 12:21:35.890628 6560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance:
uuid: "93af5027e9c74cf6ae5aeb5a96e71290"
format_stamp: "Formatted at 2026-09-28 12:21:35 on dist-test-slave-920p"
server_key: "1f167ef43035d23eadb715ec982d42cd"
server_key_iv: "4908528f5f17875e047f457323517d37"
server_key_version: "encryptionkey@0"
I20260928 12:21:35.892158 6560 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:21:35.893141 6574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:35.893359 6560 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:21:35.893455 6560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-1/wal
uuid: "93af5027e9c74cf6ae5aeb5a96e71290"
format_stamp: "Formatted at 2026-09-28 12:21:35 on dist-test-slave-920p"
server_key: "1f167ef43035d23eadb715ec982d42cd"
server_key_iv: "4908528f5f17875e047f457323517d37"
server_key_version: "encryptionkey@0"
I20260928 12:21:35.893522 6560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:35.918569 6560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:35.920755 6560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:35.920892 6560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:35.921154 6560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:21:35.921543 6560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:35.921579 6560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:35.921603 6560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:35.921620 6560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:35.928130 6560 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.2:38191
I20260928 12:21:35.928172 6687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.2:38191 every 8 connection(s)
I20260928 12:21:35.928479 6560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
I20260928 12:21:35.931257 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 6560
I20260928 12:21:35.931339 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance
I20260928 12:21:35.931533 2272 external_mini_cluster.cc:1428] Setting key 353c54de1a1ff814879d3fc6b20768e7
Sep 28 12:21:35 dist-test-slave-920p krb5kdc[6321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598095, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:35.938905 6405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:35.929326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:47505 (local address 127.2.56.62:39411)
0928 12:21:35.929386 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:35.929389 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:35.929631 (+ 242us) server_negotiation.cc:407] Connection header received
0928 12:21:35.930362 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:35.930364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:35.930379 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:35.930438 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:35.931540 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:35.932202 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:35.933346 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:35.933662 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:35.933739 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:35.936219 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:35.936220 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:35.936221 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:35.936229 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:35.937561 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:35.937951 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:35.937953 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:35.937953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:35.938005 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:35.938238 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:35.938239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:35.938240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:35.938366 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:35.938426 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:35.938706 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 12:21:35.938791 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 12:21:35.939097 6690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:35.929300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39411 (local address 127.2.56.2:47505)
0928 12:21:35.929474 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:35.929526 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:35.930237 (+ 711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:35.930510 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:35.930514 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:35.930809 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:35.931426 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:35.931434 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:35.932289 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:35.932290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:35.933252 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:35.933255 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:35.933749 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:35.933750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:35.933908 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:35.934463 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:35.934475 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:35.936130 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:35.937634 (+ 1504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:35.937642 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:35.937645 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:35.937886 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:35.938058 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:35.938059 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:35.938059 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:35.938183 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:35.938420 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:35.938426 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:35.938583 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 12:21:35.938678 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 12:21:35.938857 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20260928 12:21:35.939602 6688 heartbeater.cc:347] Connected to a master server at 127.2.56.62:39411
I20260928 12:21:35.939724 6688 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:35.939924 6688 heartbeater.cc:511] Master 127.2.56.62:39411 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.56.3@KRBTEST.COM; defaulting to no policy
I20260928 12:21:35.940515 6365 ts_manager.cc:194] Registered new tserver with Master: 93af5027e9c74cf6ae5aeb5a96e71290 (127.2.56.2:38191)
I20260928 12:21:35.940904 6365 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.2@KRBTEST.COM'} at 127.2.56.2:47505
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.3@KRBTEST.COM" created.
I20260928 12:21:36.761363 6555 heartbeater.cc:503] Master 127.2.56.62:39411 was elected leader, sending a full tablet report...
I20260928 12:21:36.941848 6688 heartbeater.cc:503] Master 127.2.56.62:39411 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab.
Entry for principal kudu/127.2.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab.
W20260928 12:21:37.020345 2272 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.2.56.3: real 1.088s user 0.000s sys 0.003s
I20260928 12:21:37.020793 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.3:0
--local_ip_for_outbound_sockets=127.2.56.3
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:39411
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:43307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:37.108407 6693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:37.108626 6693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:37.108649 6693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:37.109849 6693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:37.109890 6693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:37.109938 6693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.3
I20260928 12:21:37.111490 6693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:43307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:39411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.3
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:37.111732 6693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:37.111968 6693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:37.117027 6698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:37.117105 6701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:37.117247 6699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:37.117233 6693 server_base.cc:1034] running on GCE node
I20260928 12:21:37.117516 6693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:37.117774 6693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:37.118913 6693 hybrid_clock.cc:648] HybridClock initialized: now 1790598097118914 us; error 43 us; skew 500 ppm
Sep 28 12:21:37 dist-test-slave-920p krb5kdc[6321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598097, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:37.130371 6693 init.cc:377] Logged in from keytab as kudu/127.2.56.3@KRBTEST.COM (short username kudu)
I20260928 12:21:37.130766 6693 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:37.130820 6693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:37.130966 6693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:37.132153 6693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-2/data/instance:
uuid: "458d180d988b4b859cd62193719b7b22"
format_stamp: "Formatted at 2026-09-28 12:21:37 on dist-test-slave-920p"
server_key: "d44fac29d9f9094884625289f0312747"
server_key_iv: "99d53a9c59347afdd1da506f89cfaab2"
server_key_version: "encryptionkey@0"
I20260928 12:21:37.132491 6693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance:
uuid: "458d180d988b4b859cd62193719b7b22"
format_stamp: "Formatted at 2026-09-28 12:21:37 on dist-test-slave-920p"
server_key: "d44fac29d9f9094884625289f0312747"
server_key_iv: "99d53a9c59347afdd1da506f89cfaab2"
server_key_version: "encryptionkey@0"
I20260928 12:21:37.133868 6693 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:21:37.134615 6707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:37.134752 6693 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:37.134835 6693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-2/wal
uuid: "458d180d988b4b859cd62193719b7b22"
format_stamp: "Formatted at 2026-09-28 12:21:37 on dist-test-slave-920p"
server_key: "d44fac29d9f9094884625289f0312747"
server_key_iv: "99d53a9c59347afdd1da506f89cfaab2"
server_key_version: "encryptionkey@0"
I20260928 12:21:37.134902 6693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:37.153086 6693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:37.155131 6693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:37.155274 6693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:37.155510 6693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:21:37.155854 6693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:37.155889 6693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:37.155922 6693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:37.155956 6693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:37.162241 6693 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.3:42167
I20260928 12:21:37.162292 6820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.3:42167 every 8 connection(s)
I20260928 12:21:37.162576 6693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
I20260928 12:21:37.166285 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 6693
I20260928 12:21:37.166388 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance
I20260928 12:21:37.166584 2272 external_mini_cluster.cc:1428] Setting key fe658603f3d32362ae4878a3da1b0d6d
Sep 28 12:21:37 dist-test-slave-920p krb5kdc[6321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598097, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:37.172113 6824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:37.163331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:34937 (local address 127.2.56.62:39411)
0928 12:21:37.163485 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:37.163488 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:37.163581 (+ 93us) server_negotiation.cc:407] Connection header received
0928 12:21:37.164145 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:37.164146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:37.164159 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:37.164224 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:37.165211 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:37.165829 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:37.167024 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:37.167315 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:37.167383 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:37.169589 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:37.169591 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:37.169591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:37.169599 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:37.170806 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:37.171181 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:37.171182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:37.171183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:37.171248 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:37.171464 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:37.171464 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:37.171465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:37.171589 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:37.171646 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:37.171914 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 12:21:37.172007 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":71,"threads_started":1}
I20260928 12:21:37.172302 6823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:37.163292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39411 (local address 127.2.56.3:34937)
0928 12:21:37.163458 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:37.163499 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:37.164063 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:37.164287 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:37.164292 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:37.164581 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:37.165134 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:37.165141 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:37.165908 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:37.165909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:37.166947 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:37.166950 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:37.167390 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:37.167391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:37.167540 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:37.168074 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:37.168083 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:37.169499 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:37.170895 (+ 1396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:37.170902 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:37.170905 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:37.171116 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:37.171298 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:37.171299 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:37.171300 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:37.171410 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:37.171640 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:37.171646 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:37.171794 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 12:21:37.171893 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 12:21:37.172062 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20260928 12:21:37.172765 6821 heartbeater.cc:347] Connected to a master server at 127.2.56.62:39411
I20260928 12:21:37.172860 6821 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:37.173022 6821 heartbeater.cc:511] Master 127.2.56.62:39411 requested a full tablet report, sending...
I20260928 12:21:37.173480 6365 ts_manager.cc:194] Registered new tserver with Master: 458d180d988b4b859cd62193719b7b22 (127.2.56.3:42167)
I20260928 12:21:37.173864 6365 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.3@KRBTEST.COM'} at 127.2.56.3:34937
I20260928 12:21:37.178195 2272 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:21:37.185241 6824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:37.179590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59814 (local address 127.2.56.62:39411)
0928 12:21:37.179671 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:37.179673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:37.179814 (+ 141us) server_negotiation.cc:407] Connection header received
0928 12:21:37.179868 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:37.179868 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:37.179877 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:37.179914 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:37.180490 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:37.180985 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:37.181707 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:37.181893 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:37.181938 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:37.182997 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:37.182998 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:37.182999 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:37.183004 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:37.184244 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:37.184562 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:37.184562 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:37.184563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:37.184609 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:37.184733 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:37.184733 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:37.184733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:37.184876 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:37.184958 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:37.185097 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 12:21:37.185164 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 12:21:37.186833 6365 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:21:37.188651 6365 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:21:37.195670 6839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:37.191602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46374 (local address 127.2.56.1:34439)
0928 12:21:37.191735 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:37.191739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:37.192120 (+ 381us) server_negotiation.cc:407] Connection header received
0928 12:21:37.192165 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:37.192167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:37.192182 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:37.192260 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:37.192371 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:37.193505 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:37.194742 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:37.195270 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:37.195336 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:37.195403 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:37.195534 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 12:21:37.195576 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
I20260928 12:21:37.195771 6835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:37.191267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:38191 (local address 127.0.0.1:45452)
0928 12:21:37.191943 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:37.191950 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:37.191995 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:37.192371 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:37.192372 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:37.192375 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:37.192422 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:37.192423 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:37.193683 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:37.193684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:37.194545 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:37.194547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:37.195518 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:37.195519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:37.195591 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:37.195595 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:21:37.195640 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 12:21:37.195689 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":4,"thread_start_us":111,"threads_started":1}
I20260928 12:21:37.195858 6837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:37.191467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57152 (local address 127.2.56.3:42167)
0928 12:21:37.191662 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:37.191668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:37.191818 (+ 150us) server_negotiation.cc:407] Connection header received
0928 12:21:37.191919 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:37.191922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:37.191948 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:37.192031 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:37.192216 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:37.193186 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:37.194692 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:37.195389 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:37.195445 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:37.195492 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:37.195732 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 12:21:37.195773 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":79,"threads_started":1}
I20260928 12:21:37.196067 6836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:37.191266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:42167 (local address 127.0.0.1:57152)
0928 12:21:37.191705 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:37.191746 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:37.191818 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:37.192034 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:37.192038 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:37.192040 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:37.192130 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:37.192132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:37.193338 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:37.193338 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:37.194620 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:37.194623 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:37.195519 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:37.195519 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:37.195596 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:37.195599 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:21:37.195640 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 12:21:37.195690 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":3,"thread_start_us":106,"threads_started":1}
I20260928 12:21:37.195663 6834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:37.191266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:34439 (local address 127.0.0.1:46374)
0928 12:21:37.192059 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:37.192067 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:37.192112 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:37.192269 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:37.192269 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:37.192272 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:37.192320 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:37.192322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:37.193584 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:37.193585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:37.194657 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:37.194659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:37.195346 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:37.195347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:37.195432 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:37.195449 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 12:21:37.195513 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 12:21:37.195576 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"mutex_wait_us":6,"thread_start_us":132,"threads_started":1}
I20260928 12:21:37.196363 6838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:37.191530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45452 (local address 127.2.56.2:38191)
0928 12:21:37.191649 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:37.191656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:37.192164 (+ 508us) server_negotiation.cc:407] Connection header received
0928 12:21:37.192205 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:37.192207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:37.192224 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:37.192298 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:37.192494 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:37.193611 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:37.194621 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:37.195344 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:37.195413 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:37.195483 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:37.195945 (+ 462us) server_negotiation.cc:299] Negotiation successful
0928 12:21:37.195995 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":59,"threads_started":1}
I20260928 12:21:37.196856 6755 tablet_service.cc:1467] Processing CreateTablet for tablet 4712082cbeab4187a4a4fec469cf5b37 (DEFAULT_TABLE table=test-table [id=87ab59a0ec0e4f77ae524db48be1ddca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:37.196858 6489 tablet_service.cc:1467] Processing CreateTablet for tablet 4712082cbeab4187a4a4fec469cf5b37 (DEFAULT_TABLE table=test-table [id=87ab59a0ec0e4f77ae524db48be1ddca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:37.197013 6614 tablet_service.cc:1467] Processing CreateTablet for tablet 4712082cbeab4187a4a4fec469cf5b37 (DEFAULT_TABLE table=test-table [id=87ab59a0ec0e4f77ae524db48be1ddca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:37.197244 6489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4712082cbeab4187a4a4fec469cf5b37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:37.197242 6755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4712082cbeab4187a4a4fec469cf5b37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:37.197336 6614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4712082cbeab4187a4a4fec469cf5b37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:37.200093 6840 tablet_bootstrap.cc:492] T 4712082cbeab4187a4a4fec469cf5b37 P 458d180d988b4b859cd62193719b7b22: Bootstrap starting.
I20260928 12:21:37.200593 6842 tablet_bootstrap.cc:492] T 4712082cbeab4187a4a4fec469cf5b37 P 93af5027e9c74cf6ae5aeb5a96e71290: Bootstrap starting.
I20260928 12:21:37.201063 6840 tablet_bootstrap.cc:654] T 4712082cbeab4187a4a4fec469cf5b37 P 458d180d988b4b859cd62193719b7b22: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:37.201300 6842 tablet_bootstrap.cc:654] T 4712082cbeab4187a4a4fec469cf5b37 P 93af5027e9c74cf6ae5aeb5a96e71290: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:37.201431 6841 tablet_bootstrap.cc:492] T 4712082cbeab4187a4a4fec469cf5b37 P bd3ecd7b1284485989ff10cc2fe77b1f: Bootstrap starting.
I20260928 12:21:37.201629 6842 log.cc:824] T 4712082cbeab4187a4a4fec469cf5b37 P 93af5027e9c74cf6ae5aeb5a96e71290: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:37.201788 6840 log.cc:824] T 4712082cbeab4187a4a4fec469cf5b37 P 458d180d988b4b859cd62193719b7b22: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:37.202057 6841 tablet_bootstrap.cc:654] T 4712082cbeab4187a4a4fec469cf5b37 P bd3ecd7b1284485989ff10cc2fe77b1f: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:37.202356 6841 log.cc:824] T 4712082cbeab4187a4a4fec469cf5b37 P bd3ecd7b1284485989ff10cc2fe77b1f: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:37.202558 6840 tablet_bootstrap.cc:492] T 4712082cbeab4187a4a4fec469cf5b37 P 458d180d988b4b859cd62193719b7b22: No bootstrap required, opened a new log
I20260928 12:21:37.202589 6842 tablet_bootstrap.cc:492] T 4712082cbeab4187a4a4fec469cf5b37 P 93af5027e9c74cf6ae5aeb5a96e71290: No bootstrap required, opened a new log
I20260928 12:21:37.202634 6840 ts_tablet_manager.cc:1397] T 4712082cbeab4187a4a4fec469cf5b37 P 458d180d988b4b859cd62193719b7b22: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260928 12:21:37.202648 6842 ts_tablet_manager.cc:1397] T 4712082cbeab4187a4a4fec469cf5b37 P 93af5027e9c74cf6ae5aeb5a96e71290: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 12:21:37.203145 6841 tablet_bootstrap.cc:492] T 4712082cbeab4187a4a4fec469cf5b37 P bd3ecd7b1284485989ff10cc2fe77b1f: No bootstrap required, opened a new log
I20260928 12:21:37.203220 6841 ts_tablet_manager.cc:1397] T 4712082cbeab4187a4a4fec469cf5b37 P bd3ecd7b1284485989ff10cc2fe77b1f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:21:37.204320 6840 raft_consensus.cc:348] T 4712082cbeab4187a4a4fec469cf5b37 P 458d180d988b4b859cd62193719b7b22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458d180d988b4b859cd62193719b7b22" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 42167 } } peers { permanent_uuid: "93af5027e9c74cf6ae5aeb5a96e71290" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 38191 } } peers { permanent_uuid: "bd3ecd7b1284485989ff10cc2fe77b1f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 34439 } }
I20260928 12:21:37.204499 6840 raft_consensus.cc:374] T 4712082cbeab4187a4a4fec469cf5b37 P 458d180d988b4b859cd62193719b7b22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:37.204542 6840 raft_consensus.cc:729] T 4712082cbeab4187a4a4fec469cf5b37 P 458d180d988b4b859cd62193719b7b22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 458d180d988b4b859cd62193719b7b22, State: Initialized, Role: FOLLOWER
I20260928 12:21:37.204665 6840 consensus_queue.cc:260] T 4712082cbeab4187a4a4fec469cf5b37 P 458d180d988b4b859cd62193719b7b22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458d180d988b4b859cd62193719b7b22" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 42167 } } peers { permanent_uuid: "93af5027e9c74cf6ae5aeb5a96e71290" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 38191 } } peers { permanent_uuid: "bd3ecd7b1284485989ff10cc2fe77b1f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 34439 } }
I20260928 12:21:37.204933 6840 ts_tablet_manager.cc:1428] T 4712082cbeab4187a4a4fec469cf5b37 P 458d180d988b4b859cd62193719b7b22: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 12:21:37.204943 6821 heartbeater.cc:503] Master 127.2.56.62:39411 was elected leader, sending a full tablet report...
I20260928 12:21:37.205806 6842 raft_consensus.cc:348] T 4712082cbeab4187a4a4fec469cf5b37 P 93af5027e9c74cf6ae5aeb5a96e71290 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458d180d988b4b859cd62193719b7b22" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 42167 } } peers { permanent_uuid: "93af5027e9c74cf6ae5aeb5a96e71290" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 38191 } } peers { permanent_uuid: "bd3ecd7b1284485989ff10cc2fe77b1f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 34439 } }
I20260928 12:21:37.205858 6841 raft_consensus.cc:348] T 4712082cbeab4187a4a4fec469cf5b37 P bd3ecd7b1284485989ff10cc2fe77b1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458d180d988b4b859cd62193719b7b22" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 42167 } } peers { permanent_uuid: "93af5027e9c74cf6ae5aeb5a96e71290" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 38191 } } peers { permanent_uuid: "bd3ecd7b1284485989ff10cc2fe77b1f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 34439 } }
I20260928 12:21:37.205996 6841 raft_consensus.cc:374] T 4712082cbeab4187a4a4fec469cf5b37 P bd3ecd7b1284485989ff10cc2fe77b1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:37.206027 6842 raft_consensus.cc:374] T 4712082cbeab4187a4a4fec469cf5b37 P 93af5027e9c74cf6ae5aeb5a96e71290 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:37.206054 6841 raft_consensus.cc:729] T 4712082cbeab4187a4a4fec469cf5b37 P bd3ecd7b1284485989ff10cc2fe77b1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd3ecd7b1284485989ff10cc2fe77b1f, State: Initialized, Role: FOLLOWER
I20260928 12:21:37.206085 6842 raft_consensus.cc:729] T 4712082cbeab4187a4a4fec469cf5b37 P 93af5027e9c74cf6ae5aeb5a96e71290 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93af5027e9c74cf6ae5aeb5a96e71290, State: Initialized, Role: FOLLOWER
I20260928 12:21:37.206215 6841 consensus_queue.cc:260] T 4712082cbeab4187a4a4fec469cf5b37 P bd3ecd7b1284485989ff10cc2fe77b1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458d180d988b4b859cd62193719b7b22" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 42167 } } peers { permanent_uuid: "93af5027e9c74cf6ae5aeb5a96e71290" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 38191 } } peers { permanent_uuid: "bd3ecd7b1284485989ff10cc2fe77b1f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 34439 } }
I20260928 12:21:37.206215 6842 consensus_queue.cc:260] T 4712082cbeab4187a4a4fec469cf5b37 P 93af5027e9c74cf6ae5aeb5a96e71290 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458d180d988b4b859cd62193719b7b22" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 42167 } } peers { permanent_uuid: "93af5027e9c74cf6ae5aeb5a96e71290" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 38191 } } peers { permanent_uuid: "bd3ecd7b1284485989ff10cc2fe77b1f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 34439 } }
I20260928 12:21:37.206524 6842 ts_tablet_manager.cc:1428] T 4712082cbeab4187a4a4fec469cf5b37 P 93af5027e9c74cf6ae5aeb5a96e71290: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260928 12:21:37.206524 6841 ts_tablet_manager.cc:1428] T 4712082cbeab4187a4a4fec469cf5b37 P bd3ecd7b1284485989ff10cc2fe77b1f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 12:21:37.223784 6848 raft_consensus.cc:482] T 4712082cbeab4187a4a4fec469cf5b37 P bd3ecd7b1284485989ff10cc2fe77b1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:21:37.223951 6848 raft_consensus.cc:504] T 4712082cbeab4187a4a4fec469cf5b37 P bd3ecd7b1284485989ff10cc2fe77b1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458d180d988b4b859cd62193719b7b22" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 42167 } } peers { permanent_uuid: "93af5027e9c74cf6ae5aeb5a96e71290" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 38191 } } peers { permanent_uuid: "bd3ecd7b1284485989ff10cc2fe77b1f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 34439 } }
I20260928 12:21:37.224422 6848 leader_election.cc:288] T 4712082cbeab4187a4a4fec469cf5b37 P bd3ecd7b1284485989ff10cc2fe77b1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 458d180d988b4b859cd62193719b7b22 (127.2.56.3:42167), 93af5027e9c74cf6ae5aeb5a96e71290 (127.2.56.2:38191)
I20260928 12:21:37.227542 6838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:37.224707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:40325 (local address 127.2.56.2:38191)
0928 12:21:37.224849 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:37.224851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:37.224901 (+ 50us) server_negotiation.cc:407] Connection header received
0928 12:21:37.225082 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:37.225083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:37.225093 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:37.225129 (+ 36us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:37.225299 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:37.225782 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:37.226862 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:37.227220 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:37.227280 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:37.227293 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:37.227422 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 12:21:37.227459 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260928 12:21:37.227581 6849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:37.224608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:38191 (local address 127.2.56.1:40325)
0928 12:21:37.224809 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:37.224822 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:37.224899 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:37.225160 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:37.225161 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:37.225162 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:37.225219 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:37.225220 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:37.225857 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:37.225858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:37.226774 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:37.226777 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:37.227291 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:37.227292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:37.227363 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:37.227368 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:21:37.227411 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 12:21:37.227459 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":50,"thread_start_us":138,"threads_started":1}
I20260928 12:21:37.227576 6837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:37.224764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:51903 (local address 127.2.56.3:42167)
0928 12:21:37.224814 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:37.224816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:37.224889 (+ 73us) server_negotiation.cc:407] Connection header received
0928 12:21:37.224966 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:37.224967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:37.224978 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:37.225027 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:37.225233 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:37.225770 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:37.226860 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:37.227252 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:37.227310 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:37.227320 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:37.227460 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 12:21:37.227502 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:21:37.227597 6850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:37.224626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:42167 (local address 127.2.56.1:51903)
0928 12:21:37.224813 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:37.224822 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:37.224899 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:37.225060 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:37.225061 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:37.225064 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:37.225160 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:37.225162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:37.225844 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:37.225845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:37.226776 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:37.226778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:37.227316 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:37.227317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:37.227390 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:37.227393 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:21:37.227438 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 12:21:37.227484 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":34,"thread_start_us":107,"threads_started":1}
I20260928 12:21:37.227954 6775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4712082cbeab4187a4a4fec469cf5b37" candidate_uuid: "bd3ecd7b1284485989ff10cc2fe77b1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "458d180d988b4b859cd62193719b7b22" is_pre_election: true
I20260928 12:21:37.228051 6642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4712082cbeab4187a4a4fec469cf5b37" candidate_uuid: "bd3ecd7b1284485989ff10cc2fe77b1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93af5027e9c74cf6ae5aeb5a96e71290" is_pre_election: true
I20260928 12:21:37.228142 6775 raft_consensus.cc:2454] T 4712082cbeab4187a4a4fec469cf5b37 P 458d180d988b4b859cd62193719b7b22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd3ecd7b1284485989ff10cc2fe77b1f in term 0.
I20260928 12:21:37.228250 6642 raft_consensus.cc:2454] T 4712082cbeab4187a4a4fec469cf5b37 P 93af5027e9c74cf6ae5aeb5a96e71290 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd3ecd7b1284485989ff10cc2fe77b1f in term 0.
I20260928 12:21:37.228391 6443 leader_election.cc:302] T 4712082cbeab4187a4a4fec469cf5b37 P bd3ecd7b1284485989ff10cc2fe77b1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 458d180d988b4b859cd62193719b7b22, bd3ecd7b1284485989ff10cc2fe77b1f; no voters:
I20260928 12:21:37.228600 6848 raft_consensus.cc:2781] T 4712082cbeab4187a4a4fec469cf5b37 P bd3ecd7b1284485989ff10cc2fe77b1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:21:37.228641 6848 raft_consensus.cc:482] T 4712082cbeab4187a4a4fec469cf5b37 P bd3ecd7b1284485989ff10cc2fe77b1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:21:37.228663 6848 raft_consensus.cc:3037] T 4712082cbeab4187a4a4fec469cf5b37 P bd3ecd7b1284485989ff10cc2fe77b1f [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:37.229506 6848 raft_consensus.cc:504] T 4712082cbeab4187a4a4fec469cf5b37 P bd3ecd7b1284485989ff10cc2fe77b1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458d180d988b4b859cd62193719b7b22" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 42167 } } peers { permanent_uuid: "93af5027e9c74cf6ae5aeb5a96e71290" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 38191 } } peers { permanent_uuid: "bd3ecd7b1284485989ff10cc2fe77b1f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 34439 } }
I20260928 12:21:37.229637 6848 leader_election.cc:288] T 4712082cbeab4187a4a4fec469cf5b37 P bd3ecd7b1284485989ff10cc2fe77b1f [CANDIDATE]: Term 1 election: Requested vote from peers 458d180d988b4b859cd62193719b7b22 (127.2.56.3:42167), 93af5027e9c74cf6ae5aeb5a96e71290 (127.2.56.2:38191)
I20260928 12:21:37.229772 6642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4712082cbeab4187a4a4fec469cf5b37" candidate_uuid: "bd3ecd7b1284485989ff10cc2fe77b1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93af5027e9c74cf6ae5aeb5a96e71290"
I20260928 12:21:37.229770 6775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4712082cbeab4187a4a4fec469cf5b37" candidate_uuid: "bd3ecd7b1284485989ff10cc2fe77b1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "458d180d988b4b859cd62193719b7b22"
I20260928 12:21:37.229842 6642 raft_consensus.cc:3037] T 4712082cbeab4187a4a4fec469cf5b37 P 93af5027e9c74cf6ae5aeb5a96e71290 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:37.229844 6775 raft_consensus.cc:3037] T 4712082cbeab4187a4a4fec469cf5b37 P 458d180d988b4b859cd62193719b7b22 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:37.230583 6642 raft_consensus.cc:2454] T 4712082cbeab4187a4a4fec469cf5b37 P 93af5027e9c74cf6ae5aeb5a96e71290 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd3ecd7b1284485989ff10cc2fe77b1f in term 1.
I20260928 12:21:37.230650 6775 raft_consensus.cc:2454] T 4712082cbeab4187a4a4fec469cf5b37 P 458d180d988b4b859cd62193719b7b22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd3ecd7b1284485989ff10cc2fe77b1f in term 1.
I20260928 12:21:37.230749 6444 leader_election.cc:302] T 4712082cbeab4187a4a4fec469cf5b37 P bd3ecd7b1284485989ff10cc2fe77b1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93af5027e9c74cf6ae5aeb5a96e71290, bd3ecd7b1284485989ff10cc2fe77b1f; no voters:
I20260928 12:21:37.230851 6848 raft_consensus.cc:2781] T 4712082cbeab4187a4a4fec469cf5b37 P bd3ecd7b1284485989ff10cc2fe77b1f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:21:37.231033 6848 raft_consensus.cc:686] T 4712082cbeab4187a4a4fec469cf5b37 P bd3ecd7b1284485989ff10cc2fe77b1f [term 1 LEADER]: Becoming Leader. State: Replica: bd3ecd7b1284485989ff10cc2fe77b1f, State: Running, Role: LEADER
I20260928 12:21:37.231134 6848 consensus_queue.cc:237] T 4712082cbeab4187a4a4fec469cf5b37 P bd3ecd7b1284485989ff10cc2fe77b1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458d180d988b4b859cd62193719b7b22" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 42167 } } peers { permanent_uuid: "93af5027e9c74cf6ae5aeb5a96e71290" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 38191 } } peers { permanent_uuid: "bd3ecd7b1284485989ff10cc2fe77b1f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 34439 } }
I20260928 12:21:37.232061 6364 catalog_manager.cc:5495] T 4712082cbeab4187a4a4fec469cf5b37 P bd3ecd7b1284485989ff10cc2fe77b1f reported cstate change: term changed from 0 to 1, leader changed from <none> to bd3ecd7b1284485989ff10cc2fe77b1f (127.2.56.1). New cstate: current_term: 1 leader_uuid: "bd3ecd7b1284485989ff10cc2fe77b1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458d180d988b4b859cd62193719b7b22" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 42167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93af5027e9c74cf6ae5aeb5a96e71290" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 38191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd3ecd7b1284485989ff10cc2fe77b1f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 34439 } health_report { overall_health: HEALTHY } } }
I20260928 12:21:37.242625 6839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:37.240354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46378 (local address 127.2.56.1:34439)
0928 12:21:37.240405 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:37.240407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:37.240415 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:21:37.240452 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:37.240453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:37.240473 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:37.240537 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:21:37.240655 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:37.241158 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:37.241908 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:37.242092 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:37.242137 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:37.242295 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:21:37.242429 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:21:37.242503 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:37.242515 (+ 12us) server_negotiation.cc:299] Negotiation successful
0928 12:21:37.242536 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 12:21:37.244660 6642 raft_consensus.cc:1261] T 4712082cbeab4187a4a4fec469cf5b37 P 93af5027e9c74cf6ae5aeb5a96e71290 [term 1 FOLLOWER]: Refusing update from remote peer bd3ecd7b1284485989ff10cc2fe77b1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:21:37.244660 6775 raft_consensus.cc:1261] T 4712082cbeab4187a4a4fec469cf5b37 P 458d180d988b4b859cd62193719b7b22 [term 1 FOLLOWER]: Refusing update from remote peer bd3ecd7b1284485989ff10cc2fe77b1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:21:37.244982 6848 consensus_queue.cc:1035] T 4712082cbeab4187a4a4fec469cf5b37 P bd3ecd7b1284485989ff10cc2fe77b1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "458d180d988b4b859cd62193719b7b22" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 42167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:21:37.245124 6851 consensus_queue.cc:1035] T 4712082cbeab4187a4a4fec469cf5b37 P bd3ecd7b1284485989ff10cc2fe77b1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "93af5027e9c74cf6ae5aeb5a96e71290" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 38191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:21:37.247629 6854 mvcc.cc:205] Tried to move back new op lower bound from 7334289806311297024 to 7334289806259273728. Current Snapshot: MvccSnapshot[applied={T|T < 7334289806311297024}]
I20260928 12:21:37.248340 6855 mvcc.cc:205] Tried to move back new op lower bound from 7334289806311297024 to 7334289806259273728. Current Snapshot: MvccSnapshot[applied={T|T < 7334289806311297024}]
I20260928 12:21:37.248735 6856 mvcc.cc:205] Tried to move back new op lower bound from 7334289806311297024 to 7334289806259273728. Current Snapshot: MvccSnapshot[applied={T|T < 7334289806311297024}]
I20260928 12:21:37.250180 6364 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:21:37.250289 6364 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:21:37.251046 6364 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6a3f307f8614f1ebeee387e3c1a34c9: Sending DeleteTablet for 3 replicas of tablet 4712082cbeab4187a4a4fec469cf5b37
I20260928 12:21:37.251453 6755 tablet_service.cc:1514] Processing DeleteTablet for tablet 4712082cbeab4187a4a4fec469cf5b37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:37 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:57152
I20260928 12:21:37.251470 6489 tablet_service.cc:1514] Processing DeleteTablet for tablet 4712082cbeab4187a4a4fec469cf5b37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:37 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:46374
I20260928 12:21:37.251688 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 6427
I20260928 12:21:37.251803 6614 tablet_service.cc:1514] Processing DeleteTablet for tablet 4712082cbeab4187a4a4fec469cf5b37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:37 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:45452
I20260928 12:21:37.252417 6863 tablet_replica.cc:330] T 4712082cbeab4187a4a4fec469cf5b37 P 458d180d988b4b859cd62193719b7b22: stopping tablet replica
I20260928 12:21:37.252518 6863 raft_consensus.cc:2229] T 4712082cbeab4187a4a4fec469cf5b37 P 458d180d988b4b859cd62193719b7b22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:21:37.252621 6863 raft_consensus.cc:2258] T 4712082cbeab4187a4a4fec469cf5b37 P 458d180d988b4b859cd62193719b7b22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:37.253135 6863 ts_tablet_manager.cc:1905] T 4712082cbeab4187a4a4fec469cf5b37 P 458d180d988b4b859cd62193719b7b22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:37.253437 6865 tablet_replica.cc:330] T 4712082cbeab4187a4a4fec469cf5b37 P 93af5027e9c74cf6ae5aeb5a96e71290: stopping tablet replica
I20260928 12:21:37.253579 6865 raft_consensus.cc:2229] T 4712082cbeab4187a4a4fec469cf5b37 P 93af5027e9c74cf6ae5aeb5a96e71290 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:21:37.253724 6865 raft_consensus.cc:2258] T 4712082cbeab4187a4a4fec469cf5b37 P 93af5027e9c74cf6ae5aeb5a96e71290 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:37.254227 6865 ts_tablet_manager.cc:1905] T 4712082cbeab4187a4a4fec469cf5b37 P 93af5027e9c74cf6ae5aeb5a96e71290: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:37.254842 6863 ts_tablet_manager.cc:1918] T 4712082cbeab4187a4a4fec469cf5b37 P 458d180d988b4b859cd62193719b7b22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:21:37.254920 6863 log.cc:1196] T 4712082cbeab4187a4a4fec469cf5b37 P 458d180d988b4b859cd62193719b7b22: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-2/wal/wals/4712082cbeab4187a4a4fec469cf5b37
I20260928 12:21:37.255158 6863 ts_tablet_manager.cc:1939] T 4712082cbeab4187a4a4fec469cf5b37 P 458d180d988b4b859cd62193719b7b22: Deleting consensus metadata
I20260928 12:21:37.255579 6350 catalog_manager.cc:4841] TS 458d180d988b4b859cd62193719b7b22 (127.2.56.3:42167): tablet 4712082cbeab4187a4a4fec469cf5b37 (table test-table [id=87ab59a0ec0e4f77ae524db48be1ddca]) successfully deleted
I20260928 12:21:37.255891 6865 ts_tablet_manager.cc:1918] T 4712082cbeab4187a4a4fec469cf5b37 P 93af5027e9c74cf6ae5aeb5a96e71290: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:21:37.255970 6865 log.cc:1196] T 4712082cbeab4187a4a4fec469cf5b37 P 93af5027e9c74cf6ae5aeb5a96e71290: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598072638673-2272-0/minicluster-data/ts-1/wal/wals/4712082cbeab4187a4a4fec469cf5b37
I20260928 12:21:37.256184 6865 ts_tablet_manager.cc:1939] T 4712082cbeab4187a4a4fec469cf5b37 P 93af5027e9c74cf6ae5aeb5a96e71290: Deleting consensus metadata
I20260928 12:21:37.256795 6351 catalog_manager.cc:4841] TS 93af5027e9c74cf6ae5aeb5a96e71290 (127.2.56.2:38191): tablet 4712082cbeab4187a4a4fec469cf5b37 (table test-table [id=87ab59a0ec0e4f77ae524db48be1ddca]) successfully deleted
W20260928 12:21:37.257824 6349 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.56.1:34439 (error 108)
I20260928 12:21:37.258045 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 6560
I20260928 12:21:37.258810 6835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:37.258677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:34439 (local address 127.0.0.1:46386)
0928 12:21:37.258731 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:37.258773 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.1:34439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 12:21:37.258926 6349 catalog_manager.cc:4584] TS bd3ecd7b1284485989ff10cc2fe77b1f (127.2.56.1:34439): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4712082cbeab4187a4a4fec469cf5b37: Network error: Client connection negotiation failed: client connection to 127.2.56.1:34439: connect: Connection refused (error 111)
I20260928 12:21:37.263649 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 6693
I20260928 12:21:37.267817 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 6334
2026-09-28T12:21:37Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2929 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T12:21:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:21:37Z Disabled control of system clock
I20260928 12:21:37.295029 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:37697
--webserver_interface=127.2.56.62
--webserver_port=0
--builtin_ntp_servers=127.2.56.20:38047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:37697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 12:21:37.380493 6870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:37.380723 6870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:37.380744 6870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:37.381947 6870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:21:37.381990 6870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:37.382001 6870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:37.382012 6870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:21:37.382022 6870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:21:37.383334 6870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:38047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:37697
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:37697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:37.383574 6870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:37.383839 6870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:37.388815 6876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:37.388832 6875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:37.388825 6878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:37.389088 6870 server_base.cc:1034] running on GCE node
I20260928 12:21:37.389353 6870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:37.389632 6870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:37.390767 6870 hybrid_clock.cc:648] HybridClock initialized: now 1790598097390758 us; error 29 us; skew 500 ppm
I20260928 12:21:37.392105 6870 webserver.cc:460] Webserver started at http://127.2.56.62:38565/ using document root <none> and password file <none>
I20260928 12:21:37.392329 6870 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:37.392378 6870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:37.392513 6870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:37.393755 6870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "125388473761422da952096b91e36583"
format_stamp: "Formatted at 2026-09-28 12:21:37 on dist-test-slave-920p"
server_key: "f3889a4acd27e37ea3f8bb069194da13"
server_key_iv: "636b179874c12d32284312de1d7f1e5f"
server_key_version: "encryptionkey@0"
I20260928 12:21:37.394115 6870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "125388473761422da952096b91e36583"
format_stamp: "Formatted at 2026-09-28 12:21:37 on dist-test-slave-920p"
server_key: "f3889a4acd27e37ea3f8bb069194da13"
server_key_iv: "636b179874c12d32284312de1d7f1e5f"
server_key_version: "encryptionkey@0"
I20260928 12:21:37.395399 6870 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 12:21:37.396063 6884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:37.396214 6870 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:37.396296 6870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "125388473761422da952096b91e36583"
format_stamp: "Formatted at 2026-09-28 12:21:37 on dist-test-slave-920p"
server_key: "f3889a4acd27e37ea3f8bb069194da13"
server_key_iv: "636b179874c12d32284312de1d7f1e5f"
server_key_version: "encryptionkey@0"
I20260928 12:21:37.396355 6870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:37.399845 6870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:37.400121 6870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:37.400238 6870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:37.404386 6870 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:37697
I20260928 12:21:37.404418 6936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:37697 every 8 connection(s)
I20260928 12:21:37.404745 6870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:21:37.405448 6937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:37.408428 6937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 125388473761422da952096b91e36583: Bootstrap starting.
I20260928 12:21:37.409065 6937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 125388473761422da952096b91e36583: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:37.409195 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 6870
I20260928 12:21:37.409313 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:21:37.409443 6937 log.cc:824] T 00000000000000000000000000000000 P 125388473761422da952096b91e36583: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:37.409526 2272 external_mini_cluster.cc:1428] Setting key d9a2b060e70dc95489d2912cbbbef039
I20260928 12:21:37.410197 6937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 125388473761422da952096b91e36583: No bootstrap required, opened a new log
I20260928 12:21:37.412514 6937 raft_consensus.cc:348] T 00000000000000000000000000000000 P 125388473761422da952096b91e36583 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125388473761422da952096b91e36583" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 37697 } }
I20260928 12:21:37.412714 6937 raft_consensus.cc:374] T 00000000000000000000000000000000 P 125388473761422da952096b91e36583 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:37.412796 6937 raft_consensus.cc:729] T 00000000000000000000000000000000 P 125388473761422da952096b91e36583 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125388473761422da952096b91e36583, State: Initialized, Role: FOLLOWER
I20260928 12:21:37.412876 6941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:37.410100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46112 (local address 127.2.56.62:37697)
0928 12:21:37.410294 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:37.410304 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:37.410336 (+ 32us) server_negotiation.cc:407] Connection header received
0928 12:21:37.411393 (+ 1057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:37.411401 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:37.411468 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:37.411605 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:37.412407 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:37.412412 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:37.412412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:21:37.412415 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:37.412462 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:21:37.412473 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:37.412523 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:37.412620 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 12:21:37.412672 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
I20260928 12:21:37.412940 6937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 125388473761422da952096b91e36583 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125388473761422da952096b91e36583" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 37697 } }
I20260928 12:21:37.413053 6937 raft_consensus.cc:388] T 00000000000000000000000000000000 P 125388473761422da952096b91e36583 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:21:37.413098 6937 raft_consensus.cc:482] T 00000000000000000000000000000000 P 125388473761422da952096b91e36583 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:21:37.413151 6937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 125388473761422da952096b91e36583 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:37.414173 6937 raft_consensus.cc:504] T 00000000000000000000000000000000 P 125388473761422da952096b91e36583 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125388473761422da952096b91e36583" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 37697 } }
I20260928 12:21:37.414280 6937 leader_election.cc:302] T 00000000000000000000000000000000 P 125388473761422da952096b91e36583 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 125388473761422da952096b91e36583; no voters:
I20260928 12:21:37.414424 6937 leader_election.cc:288] T 00000000000000000000000000000000 P 125388473761422da952096b91e36583 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:21:37.414476 6942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 125388473761422da952096b91e36583 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:21:37.414633 6942 raft_consensus.cc:686] T 00000000000000000000000000000000 P 125388473761422da952096b91e36583 [term 1 LEADER]: Becoming Leader. State: Replica: 125388473761422da952096b91e36583, State: Running, Role: LEADER
I20260928 12:21:37.414703 6937 sys_catalog.cc:564] T 00000000000000000000000000000000 P 125388473761422da952096b91e36583 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:21:37.414728 6942 consensus_queue.cc:237] T 00000000000000000000000000000000 P 125388473761422da952096b91e36583 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125388473761422da952096b91e36583" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 37697 } }
I20260928 12:21:37.415141 6944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 125388473761422da952096b91e36583 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 125388473761422da952096b91e36583. Latest consensus state: current_term: 1 leader_uuid: "125388473761422da952096b91e36583" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125388473761422da952096b91e36583" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 37697 } } }
I20260928 12:21:37.415331 6944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 125388473761422da952096b91e36583 [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:37.415132 6943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 125388473761422da952096b91e36583 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "125388473761422da952096b91e36583" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125388473761422da952096b91e36583" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 37697 } } }
I20260928 12:21:37.415731 6951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:21:37.415766 6943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 125388473761422da952096b91e36583 [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:37.416352 6951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:21:37.417909 6951 catalog_manager.cc:1347] Generated new cluster ID: dbf503260ca34544b11c6ce291739114
I20260928 12:21:37.417968 6951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:21:37.431339 6951 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:21:37.431536 6951 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:21:37.437789 6951 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 125388473761422da952096b91e36583: Generated new TSK 0
I20260928 12:21:37.438014 6951 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:21:37.441028 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.1:0
--local_ip_for_outbound_sockets=127.2.56.1
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:37697
--builtin_ntp_servers=127.2.56.20:38047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 12:21:37.549821 6961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:37.550068 6961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:37.550100 6961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:37.551357 6961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:37.551399 6961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:37.551435 6961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.1
I20260928 12:21:37.552852 6961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:38047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:37697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.1
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:37.553110 6961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:37.553458 6961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:37.558526 6967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:37.558526 6969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:37.558526 6966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:37.558833 6961 server_base.cc:1034] running on GCE node
I20260928 12:21:37.559062 6961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:37.559311 6961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:37.560464 6961 hybrid_clock.cc:648] HybridClock initialized: now 1790598097560449 us; error 33 us; skew 500 ppm
I20260928 12:21:37.561825 6961 webserver.cc:460] Webserver started at http://127.2.56.1:33205/ using document root <none> and password file <none>
I20260928 12:21:37.562070 6961 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:37.562120 6961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:37.562263 6961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:37.563462 6961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-0/data/instance:
uuid: "a35e2f07a9e54a6cbd5bbddca8b5b611"
format_stamp: "Formatted at 2026-09-28 12:21:37 on dist-test-slave-920p"
server_key: "a3c5d51f88ba784577b95073152ecc98"
server_key_iv: "963c0a489e735194c8d65d0a63e63104"
server_key_version: "encryptionkey@0"
I20260928 12:21:37.563828 6961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance:
uuid: "a35e2f07a9e54a6cbd5bbddca8b5b611"
format_stamp: "Formatted at 2026-09-28 12:21:37 on dist-test-slave-920p"
server_key: "a3c5d51f88ba784577b95073152ecc98"
server_key_iv: "963c0a489e735194c8d65d0a63e63104"
server_key_version: "encryptionkey@0"
I20260928 12:21:37.565158 6961 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260928 12:21:37.565968 6975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:37.566167 6961 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:21:37.566252 6961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-0/wal
uuid: "a35e2f07a9e54a6cbd5bbddca8b5b611"
format_stamp: "Formatted at 2026-09-28 12:21:37 on dist-test-slave-920p"
server_key: "a3c5d51f88ba784577b95073152ecc98"
server_key_iv: "963c0a489e735194c8d65d0a63e63104"
server_key_version: "encryptionkey@0"
I20260928 12:21:37.566334 6961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:37.569823 6961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:37.570117 6961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:37.570242 6961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:37.570447 6961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:21:37.570758 6961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:37.570791 6961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:37.570820 6961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:37.570843 6961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:37.577795 6961 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.1:34373
I20260928 12:21:37.577836 7088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.1:34373 every 8 connection(s)
I20260928 12:21:37.578210 6961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
I20260928 12:21:37.581689 6941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:37.579026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:43697 (local address 127.2.56.62:37697)
0928 12:21:37.579112 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:37.579114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:37.579354 (+ 240us) server_negotiation.cc:407] Connection header received
0928 12:21:37.579966 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:37.579968 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:37.579975 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:37.580039 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:37.581451 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:37.581453 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:37.581454 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:21:37.581458 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:37.581476 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:21:37.581481 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:37.581536 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:37.581605 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 12:21:37.581619 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 12:21:37.581935 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:37.579018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:37697 (local address 127.2.56.1:43697)
0928 12:21:37.579214 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:37.579261 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:37.579808 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:37.580114 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:37.580118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:37.580128 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:37.581254 (+ 1126us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:21:37.581254 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:37.581308 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:21:37.581345 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:37.581539 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:37.581543 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:37.581546 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:21:37.581608 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 12:21:37.581668 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
I20260928 12:21:37.582484 7089 heartbeater.cc:347] Connected to a master server at 127.2.56.62:37697
I20260928 12:21:37.582598 7089 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:37.582741 7089 heartbeater.cc:511] Master 127.2.56.62:37697 requested a full tablet report, sending...
I20260928 12:21:37.583305 6901 ts_manager.cc:194] Registered new tserver with Master: a35e2f07a9e54a6cbd5bbddca8b5b611 (127.2.56.1:34373)
I20260928 12:21:37.587271 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 6961
I20260928 12:21:37.587358 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance
I20260928 12:21:37.587500 2272 external_mini_cluster.cc:1428] Setting key 89efff35a290526f5d937a593f04e6b2
I20260928 12:21:37.588469 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.2:0
--local_ip_for_outbound_sockets=127.2.56.2
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:37697
--builtin_ntp_servers=127.2.56.20:38047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 12:21:37.677678 7092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:37.677938 7092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:37.677963 7092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:37.679287 7092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:37.679339 7092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:37.679366 7092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.2
I20260928 12:21:37.681106 7092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:38047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:37697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.2
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:37.681442 7092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:37.681820 7092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:37.687357 7097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:37.687359 7100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:37.687536 7092 server_base.cc:1034] running on GCE node
W20260928 12:21:37.687418 7098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:37.687844 7092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:37.688131 7092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:37.689318 7092 hybrid_clock.cc:648] HybridClock initialized: now 1790598097689294 us; error 37 us; skew 500 ppm
I20260928 12:21:37.690898 7092 webserver.cc:460] Webserver started at http://127.2.56.2:42921/ using document root <none> and password file <none>
I20260928 12:21:37.691183 7092 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:37.691242 7092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:37.691411 7092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:37.692874 7092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-1/data/instance:
uuid: "e4ed4d22312842f6a8e41b08524345ff"
format_stamp: "Formatted at 2026-09-28 12:21:37 on dist-test-slave-920p"
server_key: "ed4d256869802fd67d0aa0ff8df771cf"
server_key_iv: "90ee99501e08e3b3c31a9d9d558217bc"
server_key_version: "encryptionkey@0"
I20260928 12:21:37.693370 7092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance:
uuid: "e4ed4d22312842f6a8e41b08524345ff"
format_stamp: "Formatted at 2026-09-28 12:21:37 on dist-test-slave-920p"
server_key: "ed4d256869802fd67d0aa0ff8df771cf"
server_key_iv: "90ee99501e08e3b3c31a9d9d558217bc"
server_key_version: "encryptionkey@0"
I20260928 12:21:37.695183 7092 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 12:21:37.696420 7106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:37.696668 7092 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:21:37.696787 7092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-1/wal
uuid: "e4ed4d22312842f6a8e41b08524345ff"
format_stamp: "Formatted at 2026-09-28 12:21:37 on dist-test-slave-920p"
server_key: "ed4d256869802fd67d0aa0ff8df771cf"
server_key_iv: "90ee99501e08e3b3c31a9d9d558217bc"
server_key_version: "encryptionkey@0"
I20260928 12:21:37.696885 7092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:37.701157 7092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:37.701587 7092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:37.701740 7092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:37.701983 7092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:21:37.702385 7092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:37.702422 7092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:37.702461 7092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:37.702481 7092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:37.709493 7092 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.2:33583
I20260928 12:21:37.709553 7219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.2:33583 every 8 connection(s)
I20260928 12:21:37.710014 7092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
I20260928 12:21:37.713982 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 7092
I20260928 12:21:37.714077 6941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:37.710942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:54371 (local address 127.2.56.62:37697)
0928 12:21:37.711036 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:37.711039 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:37.711335 (+ 296us) server_negotiation.cc:407] Connection header received
0928 12:21:37.711989 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:37.711991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:37.712000 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:37.712058 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:37.713616 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:37.713619 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:37.713619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:21:37.713625 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:37.713653 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:21:37.713658 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:37.713719 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:37.713968 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 12:21:37.713994 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 12:21:37.714120 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance
I20260928 12:21:37.714372 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:37.710919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:37697 (local address 127.2.56.2:54371)
0928 12:21:37.711142 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:37.711187 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:37.711841 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:37.712153 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:37.712158 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:37.712169 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:37.713385 (+ 1216us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:21:37.713385 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:37.713429 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:21:37.713479 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:37.713747 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:37.713756 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:37.713760 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:21:37.713961 (+ 201us) client_negotiation.cc:240] Negotiation successful
0928 12:21:37.714069 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":95,"threads_started":1}
I20260928 12:21:37.714413 2272 external_mini_cluster.cc:1428] Setting key c7670f4243aa05fc57208ad5a7dd5be5
I20260928 12:21:37.715171 7220 heartbeater.cc:347] Connected to a master server at 127.2.56.62:37697
I20260928 12:21:37.715302 7220 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:37.715538 7220 heartbeater.cc:511] Master 127.2.56.62:37697 requested a full tablet report, sending...
I20260928 12:21:37.715965 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.3:0
--local_ip_for_outbound_sockets=127.2.56.3
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:37697
--builtin_ntp_servers=127.2.56.20:38047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 12:21:37.716048 6900 ts_manager.cc:194] Registered new tserver with Master: e4ed4d22312842f6a8e41b08524345ff (127.2.56.2:33583)
W20260928 12:21:37.808360 7223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:37.808629 7223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:37.808652 7223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:37.809952 7223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:37.809994 7223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:37.810022 7223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.3
I20260928 12:21:37.811432 7223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:38047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:37697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.3
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:37.811689 7223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:37.811988 7223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:21:37.817469 7223 server_base.cc:1034] running on GCE node
W20260928 12:21:37.817437 7228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:37.817411 7231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:37.817411 7229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:37.817955 7223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:37.818244 7223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:37.819389 7223 hybrid_clock.cc:648] HybridClock initialized: now 1790598097819380 us; error 31 us; skew 500 ppm
I20260928 12:21:37.820785 7223 webserver.cc:460] Webserver started at http://127.2.56.3:43321/ using document root <none> and password file <none>
I20260928 12:21:37.821038 7223 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:37.821091 7223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:37.821228 7223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:37.822494 7223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-2/data/instance:
uuid: "8a92f10f509d400e8fbb6e15faf91fd8"
format_stamp: "Formatted at 2026-09-28 12:21:37 on dist-test-slave-920p"
server_key: "4fc9e4e8414fe88c1f97cb666f98ea46"
server_key_iv: "7ece7ead43d4295c1826560c7445e78f"
server_key_version: "encryptionkey@0"
I20260928 12:21:37.822850 7223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance:
uuid: "8a92f10f509d400e8fbb6e15faf91fd8"
format_stamp: "Formatted at 2026-09-28 12:21:37 on dist-test-slave-920p"
server_key: "4fc9e4e8414fe88c1f97cb666f98ea46"
server_key_iv: "7ece7ead43d4295c1826560c7445e78f"
server_key_version: "encryptionkey@0"
I20260928 12:21:37.824177 7223 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:21:37.824941 7237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:37.825096 7223 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:37.825178 7223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-2/wal
uuid: "8a92f10f509d400e8fbb6e15faf91fd8"
format_stamp: "Formatted at 2026-09-28 12:21:37 on dist-test-slave-920p"
server_key: "4fc9e4e8414fe88c1f97cb666f98ea46"
server_key_iv: "7ece7ead43d4295c1826560c7445e78f"
server_key_version: "encryptionkey@0"
I20260928 12:21:37.825239 7223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:37.828825 7223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:37.829101 7223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:37.829221 7223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:37.829444 7223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:21:37.829766 7223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:37.829797 7223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:37.829825 7223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:37.829849 7223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:37.836053 7223 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.3:41915
I20260928 12:21:37.836093 7350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.3:41915 every 8 connection(s)
I20260928 12:21:37.836436 7223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
I20260928 12:21:37.839389 6941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:37.837137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:54667 (local address 127.2.56.62:37697)
0928 12:21:37.837227 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:37.837230 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:37.837479 (+ 249us) server_negotiation.cc:407] Connection header received
0928 12:21:37.838011 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:37.838012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:37.838019 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:37.838070 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:37.839190 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:37.839191 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:37.839192 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:21:37.839196 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:37.839212 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:21:37.839216 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:37.839256 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:37.839315 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 12:21:37.839329 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 12:21:37.839586 7353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:37.837121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:37697 (local address 127.2.56.3:54667)
0928 12:21:37.837345 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:37.837389 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:37.837920 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:37.838135 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:37.838139 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:37.838148 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:37.839046 (+ 898us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:21:37.839047 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:37.839079 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:21:37.839112 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:37.839257 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:37.839260 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:37.839263 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:21:37.839315 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 12:21:37.839382 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":97,"threads_started":1}
I20260928 12:21:37.840050 7351 heartbeater.cc:347] Connected to a master server at 127.2.56.62:37697
I20260928 12:21:37.840137 7351 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:37.840265 7351 heartbeater.cc:511] Master 127.2.56.62:37697 requested a full tablet report, sending...
I20260928 12:21:37.840659 6900 ts_manager.cc:194] Registered new tserver with Master: 8a92f10f509d400e8fbb6e15faf91fd8 (127.2.56.3:41915)
I20260928 12:21:37.842142 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 7223
I20260928 12:21:37.842233 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance
I20260928 12:21:37.842414 2272 external_mini_cluster.cc:1428] Setting key 65e3cec26b65c2a635bde14c45b2c06c
I20260928 12:21:37.843175 2272 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:21:37.845511 6941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:37.844557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46114 (local address 127.2.56.62:37697)
0928 12:21:37.844602 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:37.844604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:37.844713 (+ 109us) server_negotiation.cc:407] Connection header received
0928 12:21:37.844758 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:37.844759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:37.844764 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:37.844807 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:37.845352 (+ 545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:37.845353 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:37.845354 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:21:37.845357 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:37.845370 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:21:37.845374 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:37.845411 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:37.845446 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 12:21:37.845457 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 12:21:37.846946 6900 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46114:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:21:37.848708 6900 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:21:37.852761 7362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:37.850918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:41915 (local address 127.0.0.1:45818)
0928 12:21:37.851250 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:37.851261 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:37.851320 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:37.851849 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:37.851851 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:37.851854 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:37.852340 (+ 486us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:21:37.852340 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:37.852356 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:21:37.852377 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:37.852592 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:37.852600 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:37.852603 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:21:37.852670 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 12:21:37.852684 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":92,"threads_started":1}
I20260928 12:21:37.852788 7363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:37.851113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45818 (local address 127.2.56.3:41915)
0928 12:21:37.851277 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:37.851284 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:37.851312 (+ 28us) server_negotiation.cc:407] Connection header received
0928 12:21:37.851393 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:37.851396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:37.851409 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:37.851497 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:37.852452 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:37.852462 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:37.852463 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:21:37.852468 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:37.852505 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:21:37.852534 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:37.852591 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:37.852685 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 12:21:37.852703 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
I20260928 12:21:37.852883 7359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:37.850894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:34373 (local address 127.0.0.1:58380)
0928 12:21:37.851042 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:37.851086 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:37.851147 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:37.851763 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:37.851770 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:37.851777 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:37.852569 (+ 792us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:21:37.852569 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:37.852575 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:21:37.852585 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:37.852782 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:37.852783 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:37.852783 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 12:21:37.852810 (+ 27us) client_negotiation.cc:240] Negotiation successful
0928 12:21:37.852820 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":99,"threads_started":1}
I20260928 12:21:37.852909 7364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:37.851355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48350 (local address 127.2.56.2:33583)
0928 12:21:37.851655 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:37.851659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:37.851669 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:21:37.851695 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:37.851697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:37.851706 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:37.851777 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:37.852543 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:37.852553 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:37.852553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:21:37.852558 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:37.852591 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:21:37.852626 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:37.852680 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:37.852798 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 12:21:37.852815 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":93,"threads_started":1}
I20260928 12:21:37.853192 7361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:37.851029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58380 (local address 127.2.56.1:34373)
0928 12:21:37.851518 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:37.851523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:37.851533 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:21:37.851559 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:37.851561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:37.851570 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:37.851737 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:37.852645 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:37.852652 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:37.852652 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:21:37.852656 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:37.852689 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:21:37.852710 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:37.853064 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:37.853087 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 12:21:37.853103 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":84,"threads_started":1}
I20260928 12:21:37.853349 7360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:37.851145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:33583 (local address 127.0.0.1:48350)
0928 12:21:37.851310 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:37.851318 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:37.851360 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:37.852085 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:37.852086 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:37.852088 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:37.852440 (+ 352us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:21:37.852441 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:37.852451 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:21:37.852463 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:37.852704 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:37.852705 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:37.852706 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 12:21:37.852749 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 12:21:37.852762 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":59,"threads_started":1}
I20260928 12:21:37.853799 7285 tablet_service.cc:1467] Processing CreateTablet for tablet bd1f39307aa046359039bfc6ccaac0d1 (DEFAULT_TABLE table=test-table [id=6a2e93ad99644d8d8c76f23d013664f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:37.853798 7023 tablet_service.cc:1467] Processing CreateTablet for tablet bd1f39307aa046359039bfc6ccaac0d1 (DEFAULT_TABLE table=test-table [id=6a2e93ad99644d8d8c76f23d013664f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:37.853988 7154 tablet_service.cc:1467] Processing CreateTablet for tablet bd1f39307aa046359039bfc6ccaac0d1 (DEFAULT_TABLE table=test-table [id=6a2e93ad99644d8d8c76f23d013664f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:37.854210 7285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd1f39307aa046359039bfc6ccaac0d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:37.854210 7023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd1f39307aa046359039bfc6ccaac0d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:37.854357 7154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd1f39307aa046359039bfc6ccaac0d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:37.857194 7365 tablet_bootstrap.cc:492] T bd1f39307aa046359039bfc6ccaac0d1 P e4ed4d22312842f6a8e41b08524345ff: Bootstrap starting.
I20260928 12:21:37.857692 7367 tablet_bootstrap.cc:492] T bd1f39307aa046359039bfc6ccaac0d1 P a35e2f07a9e54a6cbd5bbddca8b5b611: Bootstrap starting.
I20260928 12:21:37.857977 7365 tablet_bootstrap.cc:654] T bd1f39307aa046359039bfc6ccaac0d1 P e4ed4d22312842f6a8e41b08524345ff: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:37.858333 7365 log.cc:824] T bd1f39307aa046359039bfc6ccaac0d1 P e4ed4d22312842f6a8e41b08524345ff: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:37.858448 7367 tablet_bootstrap.cc:654] T bd1f39307aa046359039bfc6ccaac0d1 P a35e2f07a9e54a6cbd5bbddca8b5b611: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:37.858742 7367 log.cc:824] T bd1f39307aa046359039bfc6ccaac0d1 P a35e2f07a9e54a6cbd5bbddca8b5b611: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:37.859087 7366 tablet_bootstrap.cc:492] T bd1f39307aa046359039bfc6ccaac0d1 P 8a92f10f509d400e8fbb6e15faf91fd8: Bootstrap starting.
I20260928 12:21:37.859136 7365 tablet_bootstrap.cc:492] T bd1f39307aa046359039bfc6ccaac0d1 P e4ed4d22312842f6a8e41b08524345ff: No bootstrap required, opened a new log
I20260928 12:21:37.859191 7365 ts_tablet_manager.cc:1397] T bd1f39307aa046359039bfc6ccaac0d1 P e4ed4d22312842f6a8e41b08524345ff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:21:37.859414 7367 tablet_bootstrap.cc:492] T bd1f39307aa046359039bfc6ccaac0d1 P a35e2f07a9e54a6cbd5bbddca8b5b611: No bootstrap required, opened a new log
I20260928 12:21:37.859474 7367 ts_tablet_manager.cc:1397] T bd1f39307aa046359039bfc6ccaac0d1 P a35e2f07a9e54a6cbd5bbddca8b5b611: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:21:37.859758 7366 tablet_bootstrap.cc:654] T bd1f39307aa046359039bfc6ccaac0d1 P 8a92f10f509d400e8fbb6e15faf91fd8: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:37.860088 7366 log.cc:824] T bd1f39307aa046359039bfc6ccaac0d1 P 8a92f10f509d400e8fbb6e15faf91fd8: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:37.861692 7365 raft_consensus.cc:348] T bd1f39307aa046359039bfc6ccaac0d1 P e4ed4d22312842f6a8e41b08524345ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92f10f509d400e8fbb6e15faf91fd8" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41915 } } peers { permanent_uuid: "a35e2f07a9e54a6cbd5bbddca8b5b611" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 34373 } } peers { permanent_uuid: "e4ed4d22312842f6a8e41b08524345ff" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 33583 } }
I20260928 12:21:37.861649 7367 raft_consensus.cc:348] T bd1f39307aa046359039bfc6ccaac0d1 P a35e2f07a9e54a6cbd5bbddca8b5b611 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92f10f509d400e8fbb6e15faf91fd8" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41915 } } peers { permanent_uuid: "a35e2f07a9e54a6cbd5bbddca8b5b611" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 34373 } } peers { permanent_uuid: "e4ed4d22312842f6a8e41b08524345ff" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 33583 } }
I20260928 12:21:37.861825 7365 raft_consensus.cc:374] T bd1f39307aa046359039bfc6ccaac0d1 P e4ed4d22312842f6a8e41b08524345ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:37.861825 7367 raft_consensus.cc:374] T bd1f39307aa046359039bfc6ccaac0d1 P a35e2f07a9e54a6cbd5bbddca8b5b611 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:37.861860 7367 raft_consensus.cc:729] T bd1f39307aa046359039bfc6ccaac0d1 P a35e2f07a9e54a6cbd5bbddca8b5b611 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a35e2f07a9e54a6cbd5bbddca8b5b611, State: Initialized, Role: FOLLOWER
I20260928 12:21:37.861858 7365 raft_consensus.cc:729] T bd1f39307aa046359039bfc6ccaac0d1 P e4ed4d22312842f6a8e41b08524345ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4ed4d22312842f6a8e41b08524345ff, State: Initialized, Role: FOLLOWER
I20260928 12:21:37.861937 7366 tablet_bootstrap.cc:492] T bd1f39307aa046359039bfc6ccaac0d1 P 8a92f10f509d400e8fbb6e15faf91fd8: No bootstrap required, opened a new log
I20260928 12:21:37.861992 7366 ts_tablet_manager.cc:1397] T bd1f39307aa046359039bfc6ccaac0d1 P 8a92f10f509d400e8fbb6e15faf91fd8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 12:21:37.862013 7365 consensus_queue.cc:260] T bd1f39307aa046359039bfc6ccaac0d1 P e4ed4d22312842f6a8e41b08524345ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92f10f509d400e8fbb6e15faf91fd8" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41915 } } peers { permanent_uuid: "a35e2f07a9e54a6cbd5bbddca8b5b611" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 34373 } } peers { permanent_uuid: "e4ed4d22312842f6a8e41b08524345ff" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 33583 } }
I20260928 12:21:37.862015 7367 consensus_queue.cc:260] T bd1f39307aa046359039bfc6ccaac0d1 P a35e2f07a9e54a6cbd5bbddca8b5b611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92f10f509d400e8fbb6e15faf91fd8" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41915 } } peers { permanent_uuid: "a35e2f07a9e54a6cbd5bbddca8b5b611" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 34373 } } peers { permanent_uuid: "e4ed4d22312842f6a8e41b08524345ff" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 33583 } }
I20260928 12:21:37.862296 7367 ts_tablet_manager.cc:1428] T bd1f39307aa046359039bfc6ccaac0d1 P a35e2f07a9e54a6cbd5bbddca8b5b611: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 12:21:37.862296 7365 ts_tablet_manager.cc:1428] T bd1f39307aa046359039bfc6ccaac0d1 P e4ed4d22312842f6a8e41b08524345ff: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:21:37.862435 7220 heartbeater.cc:503] Master 127.2.56.62:37697 was elected leader, sending a full tablet report...
I20260928 12:21:37.862454 7089 heartbeater.cc:503] Master 127.2.56.62:37697 was elected leader, sending a full tablet report...
I20260928 12:21:37.864367 7366 raft_consensus.cc:348] T bd1f39307aa046359039bfc6ccaac0d1 P 8a92f10f509d400e8fbb6e15faf91fd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92f10f509d400e8fbb6e15faf91fd8" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41915 } } peers { permanent_uuid: "a35e2f07a9e54a6cbd5bbddca8b5b611" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 34373 } } peers { permanent_uuid: "e4ed4d22312842f6a8e41b08524345ff" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 33583 } }
I20260928 12:21:37.864507 7366 raft_consensus.cc:374] T bd1f39307aa046359039bfc6ccaac0d1 P 8a92f10f509d400e8fbb6e15faf91fd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:37.864529 7366 raft_consensus.cc:729] T bd1f39307aa046359039bfc6ccaac0d1 P 8a92f10f509d400e8fbb6e15faf91fd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a92f10f509d400e8fbb6e15faf91fd8, State: Initialized, Role: FOLLOWER
I20260928 12:21:37.864614 7366 consensus_queue.cc:260] T bd1f39307aa046359039bfc6ccaac0d1 P 8a92f10f509d400e8fbb6e15faf91fd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92f10f509d400e8fbb6e15faf91fd8" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41915 } } peers { permanent_uuid: "a35e2f07a9e54a6cbd5bbddca8b5b611" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 34373 } } peers { permanent_uuid: "e4ed4d22312842f6a8e41b08524345ff" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 33583 } }
I20260928 12:21:37.864813 7366 ts_tablet_manager.cc:1428] T bd1f39307aa046359039bfc6ccaac0d1 P 8a92f10f509d400e8fbb6e15faf91fd8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:21:37.864890 7351 heartbeater.cc:503] Master 127.2.56.62:37697 was elected leader, sending a full tablet report...
W20260928 12:21:37.960692 7221 tablet.cc:2369] T bd1f39307aa046359039bfc6ccaac0d1 P e4ed4d22312842f6a8e41b08524345ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:21:38.038609 7371 raft_consensus.cc:482] T bd1f39307aa046359039bfc6ccaac0d1 P e4ed4d22312842f6a8e41b08524345ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:21:38.038765 7371 raft_consensus.cc:504] T bd1f39307aa046359039bfc6ccaac0d1 P e4ed4d22312842f6a8e41b08524345ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92f10f509d400e8fbb6e15faf91fd8" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41915 } } peers { permanent_uuid: "a35e2f07a9e54a6cbd5bbddca8b5b611" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 34373 } } peers { permanent_uuid: "e4ed4d22312842f6a8e41b08524345ff" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 33583 } }
I20260928 12:21:38.039237 7371 leader_election.cc:288] T bd1f39307aa046359039bfc6ccaac0d1 P e4ed4d22312842f6a8e41b08524345ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a92f10f509d400e8fbb6e15faf91fd8 (127.2.56.3:41915), a35e2f07a9e54a6cbd5bbddca8b5b611 (127.2.56.1:34373)
I20260928 12:21:38.040512 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:38.039455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:41915 (local address 127.2.56.2:48077)
0928 12:21:38.039522 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:38.039535 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:38.039578 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:38.039725 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:38.039726 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:38.039729 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:38.040193 (+ 464us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:21:38.040194 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:38.040204 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:21:38.040222 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:38.040393 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:38.040394 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:38.040395 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 12:21:38.040434 (+ 39us) client_negotiation.cc:240] Negotiation successful
0928 12:21:38.040446 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 12:21:38.040531 7363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:38.039530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:48077 (local address 127.2.56.3:41915)
0928 12:21:38.039579 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:38.039581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:38.039622 (+ 41us) server_negotiation.cc:407] Connection header received
0928 12:21:38.039652 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:38.039653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:38.039659 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:38.039706 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:38.040304 (+ 598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:38.040305 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:38.040306 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:21:38.040310 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:38.040328 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:21:38.040332 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:38.040382 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:38.040441 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 12:21:38.040456 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:21:38.040650 7374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:38.039455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:34373 (local address 127.2.56.2:48459)
0928 12:21:38.039645 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:38.039651 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:38.039699 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:38.039812 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:38.039812 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:38.039827 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:38.040309 (+ 482us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:21:38.040309 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:38.040314 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:21:38.040326 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:38.040515 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:38.040515 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:38.040516 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 12:21:38.040568 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 12:21:38.040579 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":20,"thread_start_us":109,"threads_started":1}
I20260928 12:21:38.040659 7361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:38.039530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:48459 (local address 127.2.56.1:34373)
0928 12:21:38.039602 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:38.039605 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:38.039708 (+ 103us) server_negotiation.cc:407] Connection header received
0928 12:21:38.039744 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:38.039745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:38.039750 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:38.039803 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:38.040427 (+ 624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:38.040429 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:38.040429 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:21:38.040433 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:38.040452 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:21:38.040457 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:38.040511 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:38.040574 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 12:21:38.040588 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 12:21:38.040890 7305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1f39307aa046359039bfc6ccaac0d1" candidate_uuid: "e4ed4d22312842f6a8e41b08524345ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a92f10f509d400e8fbb6e15faf91fd8" is_pre_election: true
I20260928 12:21:38.040980 7043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1f39307aa046359039bfc6ccaac0d1" candidate_uuid: "e4ed4d22312842f6a8e41b08524345ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a35e2f07a9e54a6cbd5bbddca8b5b611" is_pre_election: true
I20260928 12:21:38.041064 7305 raft_consensus.cc:2454] T bd1f39307aa046359039bfc6ccaac0d1 P 8a92f10f509d400e8fbb6e15faf91fd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4ed4d22312842f6a8e41b08524345ff in term 0.
I20260928 12:21:38.041147 7043 raft_consensus.cc:2454] T bd1f39307aa046359039bfc6ccaac0d1 P a35e2f07a9e54a6cbd5bbddca8b5b611 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4ed4d22312842f6a8e41b08524345ff in term 0.
I20260928 12:21:38.041364 7107 leader_election.cc:302] T bd1f39307aa046359039bfc6ccaac0d1 P e4ed4d22312842f6a8e41b08524345ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a92f10f509d400e8fbb6e15faf91fd8, e4ed4d22312842f6a8e41b08524345ff; no voters:
I20260928 12:21:38.041517 7371 raft_consensus.cc:2781] T bd1f39307aa046359039bfc6ccaac0d1 P e4ed4d22312842f6a8e41b08524345ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:21:38.041574 7371 raft_consensus.cc:482] T bd1f39307aa046359039bfc6ccaac0d1 P e4ed4d22312842f6a8e41b08524345ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:21:38.041591 7371 raft_consensus.cc:3037] T bd1f39307aa046359039bfc6ccaac0d1 P e4ed4d22312842f6a8e41b08524345ff [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:38.042409 7371 raft_consensus.cc:504] T bd1f39307aa046359039bfc6ccaac0d1 P e4ed4d22312842f6a8e41b08524345ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92f10f509d400e8fbb6e15faf91fd8" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41915 } } peers { permanent_uuid: "a35e2f07a9e54a6cbd5bbddca8b5b611" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 34373 } } peers { permanent_uuid: "e4ed4d22312842f6a8e41b08524345ff" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 33583 } }
I20260928 12:21:38.042536 7371 leader_election.cc:288] T bd1f39307aa046359039bfc6ccaac0d1 P e4ed4d22312842f6a8e41b08524345ff [CANDIDATE]: Term 1 election: Requested vote from peers 8a92f10f509d400e8fbb6e15faf91fd8 (127.2.56.3:41915), a35e2f07a9e54a6cbd5bbddca8b5b611 (127.2.56.1:34373)
I20260928 12:21:38.042692 7043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1f39307aa046359039bfc6ccaac0d1" candidate_uuid: "e4ed4d22312842f6a8e41b08524345ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a35e2f07a9e54a6cbd5bbddca8b5b611"
I20260928 12:21:38.042729 7305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1f39307aa046359039bfc6ccaac0d1" candidate_uuid: "e4ed4d22312842f6a8e41b08524345ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a92f10f509d400e8fbb6e15faf91fd8"
I20260928 12:21:38.042793 7305 raft_consensus.cc:3037] T bd1f39307aa046359039bfc6ccaac0d1 P 8a92f10f509d400e8fbb6e15faf91fd8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:38.042775 7043 raft_consensus.cc:3037] T bd1f39307aa046359039bfc6ccaac0d1 P a35e2f07a9e54a6cbd5bbddca8b5b611 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:38.043560 7305 raft_consensus.cc:2454] T bd1f39307aa046359039bfc6ccaac0d1 P 8a92f10f509d400e8fbb6e15faf91fd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4ed4d22312842f6a8e41b08524345ff in term 1.
I20260928 12:21:38.043673 7043 raft_consensus.cc:2454] T bd1f39307aa046359039bfc6ccaac0d1 P a35e2f07a9e54a6cbd5bbddca8b5b611 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4ed4d22312842f6a8e41b08524345ff in term 1.
I20260928 12:21:38.043715 7107 leader_election.cc:302] T bd1f39307aa046359039bfc6ccaac0d1 P e4ed4d22312842f6a8e41b08524345ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a92f10f509d400e8fbb6e15faf91fd8, e4ed4d22312842f6a8e41b08524345ff; no voters:
I20260928 12:21:38.043788 7371 raft_consensus.cc:2781] T bd1f39307aa046359039bfc6ccaac0d1 P e4ed4d22312842f6a8e41b08524345ff [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:21:38.043949 7371 raft_consensus.cc:686] T bd1f39307aa046359039bfc6ccaac0d1 P e4ed4d22312842f6a8e41b08524345ff [term 1 LEADER]: Becoming Leader. State: Replica: e4ed4d22312842f6a8e41b08524345ff, State: Running, Role: LEADER
I20260928 12:21:38.044062 7371 consensus_queue.cc:237] T bd1f39307aa046359039bfc6ccaac0d1 P e4ed4d22312842f6a8e41b08524345ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92f10f509d400e8fbb6e15faf91fd8" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41915 } } peers { permanent_uuid: "a35e2f07a9e54a6cbd5bbddca8b5b611" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 34373 } } peers { permanent_uuid: "e4ed4d22312842f6a8e41b08524345ff" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 33583 } }
I20260928 12:21:38.044975 6900 catalog_manager.cc:5495] T bd1f39307aa046359039bfc6ccaac0d1 P e4ed4d22312842f6a8e41b08524345ff reported cstate change: term changed from 0 to 1, leader changed from <none> to e4ed4d22312842f6a8e41b08524345ff (127.2.56.2). New cstate: current_term: 1 leader_uuid: "e4ed4d22312842f6a8e41b08524345ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92f10f509d400e8fbb6e15faf91fd8" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a35e2f07a9e54a6cbd5bbddca8b5b611" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 34373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4ed4d22312842f6a8e41b08524345ff" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 33583 } health_report { overall_health: HEALTHY } } }
W20260928 12:21:38.079031 7090 tablet.cc:2369] T bd1f39307aa046359039bfc6ccaac0d1 P a35e2f07a9e54a6cbd5bbddca8b5b611: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:21:38.084611 7364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:38.083647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48352 (local address 127.2.56.2:33583)
0928 12:21:38.083712 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:38.083714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:38.083725 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:21:38.083779 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:38.083780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:38.083795 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:38.083842 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:38.084373 (+ 531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:38.084374 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:38.084375 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:21:38.084379 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:38.084397 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:21:38.084402 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:38.084449 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:38.084508 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 12:21:38.084521 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 12:21:38.086719 7043 raft_consensus.cc:1261] T bd1f39307aa046359039bfc6ccaac0d1 P a35e2f07a9e54a6cbd5bbddca8b5b611 [term 1 FOLLOWER]: Refusing update from remote peer e4ed4d22312842f6a8e41b08524345ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:21:38.086752 7305 raft_consensus.cc:1261] T bd1f39307aa046359039bfc6ccaac0d1 P 8a92f10f509d400e8fbb6e15faf91fd8 [term 1 FOLLOWER]: Refusing update from remote peer e4ed4d22312842f6a8e41b08524345ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260928 12:21:38.086967 7352 tablet.cc:2369] T bd1f39307aa046359039bfc6ccaac0d1 P 8a92f10f509d400e8fbb6e15faf91fd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:21:38.087150 7371 consensus_queue.cc:1035] T bd1f39307aa046359039bfc6ccaac0d1 P e4ed4d22312842f6a8e41b08524345ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a92f10f509d400e8fbb6e15faf91fd8" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:21:38.087334 7375 consensus_queue.cc:1035] T bd1f39307aa046359039bfc6ccaac0d1 P e4ed4d22312842f6a8e41b08524345ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a35e2f07a9e54a6cbd5bbddca8b5b611" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 34373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:21:38.090222 7380 mvcc.cc:205] Tried to move back new op lower bound from 7334289809760239616 to 7334289809589088256. Current Snapshot: MvccSnapshot[applied={T|T < 7334289809760239616}]
I20260928 12:21:38.090667 7379 mvcc.cc:205] Tried to move back new op lower bound from 7334289809760239616 to 7334289809589088256. Current Snapshot: MvccSnapshot[applied={T|T < 7334289809760239616}]
I20260928 12:21:38.091915 6900 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:21:38.092023 6900 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:21:38.092835 6900 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 125388473761422da952096b91e36583: Sending DeleteTablet for 3 replicas of tablet bd1f39307aa046359039bfc6ccaac0d1
I20260928 12:21:38.093168 7285 tablet_service.cc:1514] Processing DeleteTablet for tablet bd1f39307aa046359039bfc6ccaac0d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:38 UTC) from {username='slave'} at 127.0.0.1:45818
I20260928 12:21:38.093218 7023 tablet_service.cc:1514] Processing DeleteTablet for tablet bd1f39307aa046359039bfc6ccaac0d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:38 UTC) from {username='slave'} at 127.0.0.1:58380
I20260928 12:21:38.093466 7387 tablet_replica.cc:330] T bd1f39307aa046359039bfc6ccaac0d1 P 8a92f10f509d400e8fbb6e15faf91fd8: stopping tablet replica
I20260928 12:21:38.093508 7154 tablet_service.cc:1514] Processing DeleteTablet for tablet bd1f39307aa046359039bfc6ccaac0d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:38 UTC) from {username='slave'} at 127.0.0.1:48350
I20260928 12:21:38.093608 7387 raft_consensus.cc:2229] T bd1f39307aa046359039bfc6ccaac0d1 P 8a92f10f509d400e8fbb6e15faf91fd8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:21:38.093727 7387 raft_consensus.cc:2258] T bd1f39307aa046359039bfc6ccaac0d1 P 8a92f10f509d400e8fbb6e15faf91fd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:38.093881 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 6961
I20260928 12:21:38.094161 7387 ts_tablet_manager.cc:1905] T bd1f39307aa046359039bfc6ccaac0d1 P 8a92f10f509d400e8fbb6e15faf91fd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:38.095534 7389 tablet_replica.cc:330] T bd1f39307aa046359039bfc6ccaac0d1 P e4ed4d22312842f6a8e41b08524345ff: stopping tablet replica
I20260928 12:21:38.095628 7389 raft_consensus.cc:2229] T bd1f39307aa046359039bfc6ccaac0d1 P e4ed4d22312842f6a8e41b08524345ff [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:21:38.096493 7387 ts_tablet_manager.cc:1918] T bd1f39307aa046359039bfc6ccaac0d1 P 8a92f10f509d400e8fbb6e15faf91fd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:21:38.096585 7387 log.cc:1196] T bd1f39307aa046359039bfc6ccaac0d1 P 8a92f10f509d400e8fbb6e15faf91fd8: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598072638673-2272-0/minicluster-data/ts-2/wal/wals/bd1f39307aa046359039bfc6ccaac0d1
I20260928 12:21:38.096982 7387 ts_tablet_manager.cc:1939] T bd1f39307aa046359039bfc6ccaac0d1 P 8a92f10f509d400e8fbb6e15faf91fd8: Deleting consensus metadata
I20260928 12:21:38.098129 6885 catalog_manager.cc:4841] TS 8a92f10f509d400e8fbb6e15faf91fd8 (127.2.56.3:41915): tablet bd1f39307aa046359039bfc6ccaac0d1 (table test-table [id=6a2e93ad99644d8d8c76f23d013664f5]) successfully deleted
I20260928 12:21:38.099251 7389 raft_consensus.cc:2258] T bd1f39307aa046359039bfc6ccaac0d1 P e4ed4d22312842f6a8e41b08524345ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:38.099865 7389 ts_tablet_manager.cc:1905] T bd1f39307aa046359039bfc6ccaac0d1 P e4ed4d22312842f6a8e41b08524345ff: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 12:21:38.101331 6888 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.56.1:34373 (error 108)
I20260928 12:21:38.101910 7389 ts_tablet_manager.cc:1918] T bd1f39307aa046359039bfc6ccaac0d1 P e4ed4d22312842f6a8e41b08524345ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:21:38.102006 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 7092
I20260928 12:21:38.105635 7360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:38.105481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:34373 (local address 127.0.0.1:58396)
0928 12:21:38.105540 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:38.105594 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.1:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 12:21:38.105763 6888 catalog_manager.cc:4584] TS a35e2f07a9e54a6cbd5bbddca8b5b611 (127.2.56.1:34373): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bd1f39307aa046359039bfc6ccaac0d1: Network error: Client connection negotiation failed: client connection to 127.2.56.1:34373: connect: Connection refused (error 111)
I20260928 12:21:38.108664 7360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:38.108555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:33583 (local address 127.0.0.1:48364)
0928 12:21:38.108601 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:38.108627 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.2:33583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 12:21:38.108680 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 7223
I20260928 12:21:38.112720 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 6870
2026-09-28T12:21:38Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (844 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 12:21:38.132006 2272 mini_oidc.cc:152] Starting JWKS server
I20260928 12:21:38.133186 2272 webserver.cc:460] Webserver started at https://127.0.0.1:36075/ using document root <none> and password file <none>
I20260928 12:21:38.133394 2272 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 12:21:38.133559 2272 webserver.cc:460] Webserver started at http://127.0.0.1:38711/ using document root <none> and password file <none>
2026-09-28T12:21:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:21:38Z Disabled control of system clock
I20260928 12:21:38.140928 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:42377
--webserver_interface=127.2.56.62
--webserver_port=0
--builtin_ntp_servers=127.2.56.20:35201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:42377
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36075/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598072638673-2272-0/testchainca.cert with env {}
W20260928 12:21:38.227752 7396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:38.228022 7396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:38.228044 7396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:38.229296 7396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:21:38.229341 7396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:38.229352 7396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:38.229377 7396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:21:38.229399 7396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 12:21:38.229421 7396 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 12:21:38.229442 7396 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36075/jwks.json
I20260928 12:21:38.231087 7396 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:35201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:42377
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:42377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36075/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598072638673-2272-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:38.231364 7396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:38.231591 7396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:38.236807 7402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:38.236969 7396 server_base.cc:1034] running on GCE node
W20260928 12:21:38.236819 7401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:38.236802 7404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:38.237342 7396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:38.237668 7396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:38.238798 7396 hybrid_clock.cc:648] HybridClock initialized: now 1790598098238784 us; error 35 us; skew 500 ppm
I20260928 12:21:38.240104 7396 webserver.cc:460] Webserver started at http://127.2.56.62:46461/ using document root <none> and password file <none>
I20260928 12:21:38.240346 7396 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:38.240415 7396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:38.240545 7396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:38.241729 7396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "44a073c86d194e6a83f7c4244b5698f1"
format_stamp: "Formatted at 2026-09-28 12:21:38 on dist-test-slave-920p"
server_key: "ecb21805e33eac2aec6618da532a64b6"
server_key_iv: "ce1a1cbaa2ddaf2e723dfff094679018"
server_key_version: "encryptionkey@0"
I20260928 12:21:38.242079 7396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "44a073c86d194e6a83f7c4244b5698f1"
format_stamp: "Formatted at 2026-09-28 12:21:38 on dist-test-slave-920p"
server_key: "ecb21805e33eac2aec6618da532a64b6"
server_key_iv: "ce1a1cbaa2ddaf2e723dfff094679018"
server_key_version: "encryptionkey@0"
I20260928 12:21:38.243322 7396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:21:38.243978 7410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:38.244127 7396 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:38.244211 7396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "44a073c86d194e6a83f7c4244b5698f1"
format_stamp: "Formatted at 2026-09-28 12:21:38 on dist-test-slave-920p"
server_key: "ecb21805e33eac2aec6618da532a64b6"
server_key_iv: "ce1a1cbaa2ddaf2e723dfff094679018"
server_key_version: "encryptionkey@0"
I20260928 12:21:38.244270 7396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:38.254236 7396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:38.254559 7396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:38.254686 7396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:38.259037 7396 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:42377
I20260928 12:21:38.259075 7462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:42377 every 8 connection(s)
I20260928 12:21:38.259431 7396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:21:38.260393 7463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:38.262979 7463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44a073c86d194e6a83f7c4244b5698f1: Bootstrap starting.
I20260928 12:21:38.263612 7463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44a073c86d194e6a83f7c4244b5698f1: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:38.263926 7463 log.cc:824] T 00000000000000000000000000000000 P 44a073c86d194e6a83f7c4244b5698f1: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:38.264701 7463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44a073c86d194e6a83f7c4244b5698f1: No bootstrap required, opened a new log
I20260928 12:21:38.265128 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 7396
I20260928 12:21:38.265241 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:21:38.265465 2272 external_mini_cluster.cc:1428] Setting key c698322fc9148600c64c32f079004e9c
I20260928 12:21:38.266813 7463 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44a073c86d194e6a83f7c4244b5698f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44a073c86d194e6a83f7c4244b5698f1" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 42377 } }
I20260928 12:21:38.266969 7463 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44a073c86d194e6a83f7c4244b5698f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:38.267004 7463 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44a073c86d194e6a83f7c4244b5698f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44a073c86d194e6a83f7c4244b5698f1, State: Initialized, Role: FOLLOWER
I20260928 12:21:38.267130 7463 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44a073c86d194e6a83f7c4244b5698f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44a073c86d194e6a83f7c4244b5698f1" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 42377 } }
I20260928 12:21:38.267207 7463 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44a073c86d194e6a83f7c4244b5698f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:21:38.267237 7463 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44a073c86d194e6a83f7c4244b5698f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:21:38.267287 7463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44a073c86d194e6a83f7c4244b5698f1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:38.267831 7463 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44a073c86d194e6a83f7c4244b5698f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44a073c86d194e6a83f7c4244b5698f1" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 42377 } }
I20260928 12:21:38.267940 7463 leader_election.cc:302] T 00000000000000000000000000000000 P 44a073c86d194e6a83f7c4244b5698f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44a073c86d194e6a83f7c4244b5698f1; no voters:
I20260928 12:21:38.268095 7463 leader_election.cc:288] T 00000000000000000000000000000000 P 44a073c86d194e6a83f7c4244b5698f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:21:38.268139 7468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44a073c86d194e6a83f7c4244b5698f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:21:38.268280 7468 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44a073c86d194e6a83f7c4244b5698f1 [term 1 LEADER]: Becoming Leader. State: Replica: 44a073c86d194e6a83f7c4244b5698f1, State: Running, Role: LEADER
I20260928 12:21:38.268349 7463 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44a073c86d194e6a83f7c4244b5698f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:21:38.268378 7468 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44a073c86d194e6a83f7c4244b5698f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44a073c86d194e6a83f7c4244b5698f1" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 42377 } }
I20260928 12:21:38.268735 7468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44a073c86d194e6a83f7c4244b5698f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44a073c86d194e6a83f7c4244b5698f1. Latest consensus state: current_term: 1 leader_uuid: "44a073c86d194e6a83f7c4244b5698f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44a073c86d194e6a83f7c4244b5698f1" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 42377 } } }
I20260928 12:21:38.268803 7468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44a073c86d194e6a83f7c4244b5698f1 [sys.catalog]: This master's current role is: LEADER
W20260928 12:21:38.270201 7483 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 44a073c86d194e6a83f7c4244b5698f1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:21:38.270257 7483 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:21:38.271024 7469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44a073c86d194e6a83f7c4244b5698f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44a073c86d194e6a83f7c4244b5698f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44a073c86d194e6a83f7c4244b5698f1" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 42377 } } }
I20260928 12:21:38.271090 7469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44a073c86d194e6a83f7c4244b5698f1 [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:38.271342 7485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:21:38.271533 7485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:21:38.272137 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:38.265981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51744 (local address 127.2.56.62:42377)
0928 12:21:38.266220 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:38.266229 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:38.266255 (+ 26us) server_negotiation.cc:407] Connection header received
0928 12:21:38.266780 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:38.266787 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:38.266849 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:38.267001 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:38.267215 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:38.268824 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:38.269952 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:38.270529 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:38.270646 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:38.271398 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:38.271403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:38.271403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:21:38.271617 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:38.271664 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:21:38.271679 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:38.271734 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:38.271818 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 12:21:38.271900 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":89,"threads_started":1}
I20260928 12:21:38.273106 2272 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:21:38.273319 7485 catalog_manager.cc:1347] Generated new cluster ID: 18ecaf50c29e4cff8d1c3589fab810b2
I20260928 12:21:38.273370 7485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:21:38.282531 7485 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:21:38.283277 7485 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:21:38.287448 7485 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44a073c86d194e6a83f7c4244b5698f1: Generated new TSK 0
I20260928 12:21:38.287660 7485 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 12:21:38.312361 7493 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 12:21:38.312474 7493 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 12:21:38.312839 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:38.296791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51754 (local address 127.2.56.62:42377)
0928 12:21:38.296846 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:38.296849 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:38.296980 (+ 131us) server_negotiation.cc:407] Connection header received
0928 12:21:38.297069 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:38.297070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:38.297083 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:38.297169 (+ 86us) server_negotiation.cc:226] Negotiated authn=JWT
0928 12:21:38.297335 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:38.298165 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:38.298962 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:38.299238 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:38.299319 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:38.299517 (+ 198us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 12:21:38.312561 (+ 13044us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36075/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 12:21:38.312744 (+ 183us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51754: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36075/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":29}
W20260928 12:21:38.312906 7467 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51754: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36075/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 12:21:38.313218 7492 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:21:38.296711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:42377 (local address 127.0.0.1:51754)
0928 12:21:38.296902 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:38.296912 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:38.296979 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:38.297170 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:38.297171 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:38.297173 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 12:21:38.297237 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:38.297239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:38.298243 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:38.298244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:38.298880 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:38.298883 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:38.299318 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:38.299322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:38.299376 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:38.299400 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 12:21:38.312727 (+ 13327us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36075/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 12:21:38.312843 (+ 116us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.56.62:42377: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36075/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":84,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.2.56.62:42377: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36075/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 12:21:38.313715 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 7396
2026-09-28T12:21:38Z chronyd exiting
I20260928 12:21:38.350126 2272 test_util.cc:175] -----------------------------------------------
I20260928 12:21:38.350231 2272 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598072638673-2272-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (219 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 12:21:38.351090 2272 mini_oidc.cc:152] Starting JWKS server
I20260928 12:21:38.351837 2272 webserver.cc:460] Webserver started at https://127.0.0.1:42909/ using document root <none> and password file <none>
I20260928 12:21:38.351878 2272 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 12:21:38.352025 2272 webserver.cc:460] Webserver started at http://127.0.0.1:40337/ using document root <none> and password file <none>
2026-09-28T12:21:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:21:38Z Disabled control of system clock
I20260928 12:21:38.359628 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:36119
--webserver_interface=127.2.56.62
--webserver_port=0
--builtin_ntp_servers=127.2.56.20:37001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:36119
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42909/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598072638673-2272-0/testchainca.cert with env {}
W20260928 12:21:38.444365 7499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:38.444599 7499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:38.444621 7499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:38.445823 7499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:21:38.445863 7499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:38.445874 7499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:38.445885 7499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:21:38.445896 7499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 12:21:38.445906 7499 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 12:21:38.445916 7499 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42909/jwks.json
I20260928 12:21:38.447479 7499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:37001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:36119
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:36119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42909/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598072638673-2272-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:38.447697 7499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:38.447901 7499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:38.453022 7504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:38.452984 7507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:38.453074 7499 server_base.cc:1034] running on GCE node
W20260928 12:21:38.453034 7505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:38.453402 7499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:38.453722 7499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:38.454886 7499 hybrid_clock.cc:648] HybridClock initialized: now 1790598098454872 us; error 33 us; skew 500 ppm
I20260928 12:21:38.456223 7499 webserver.cc:460] Webserver started at http://127.2.56.62:33981/ using document root <none> and password file <none>
I20260928 12:21:38.456471 7499 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:38.456523 7499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:38.456647 7499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:38.457885 7499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "de2f4df0533c4c0dbf020214e3af2a84"
format_stamp: "Formatted at 2026-09-28 12:21:38 on dist-test-slave-920p"
server_key: "10b1fa15021b933f86a7a87f8342bb65"
server_key_iv: "7074c2a2459edd26cd1cd88cacebb5b8"
server_key_version: "encryptionkey@0"
I20260928 12:21:38.458305 7499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "de2f4df0533c4c0dbf020214e3af2a84"
format_stamp: "Formatted at 2026-09-28 12:21:38 on dist-test-slave-920p"
server_key: "10b1fa15021b933f86a7a87f8342bb65"
server_key_iv: "7074c2a2459edd26cd1cd88cacebb5b8"
server_key_version: "encryptionkey@0"
I20260928 12:21:38.459810 7499 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:21:38.460455 7513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:38.460616 7499 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:38.460693 7499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "de2f4df0533c4c0dbf020214e3af2a84"
format_stamp: "Formatted at 2026-09-28 12:21:38 on dist-test-slave-920p"
server_key: "10b1fa15021b933f86a7a87f8342bb65"
server_key_iv: "7074c2a2459edd26cd1cd88cacebb5b8"
server_key_version: "encryptionkey@0"
I20260928 12:21:38.460763 7499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:38.471148 7499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:38.471469 7499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:38.471594 7499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:38.475852 7499 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:36119
I20260928 12:21:38.475893 7565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:36119 every 8 connection(s)
I20260928 12:21:38.476269 7499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:21:38.477407 7566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:38.479939 7566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de2f4df0533c4c0dbf020214e3af2a84: Bootstrap starting.
I20260928 12:21:38.480578 7566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de2f4df0533c4c0dbf020214e3af2a84: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:38.480851 7566 log.cc:824] T 00000000000000000000000000000000 P de2f4df0533c4c0dbf020214e3af2a84: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:38.481586 7566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de2f4df0533c4c0dbf020214e3af2a84: No bootstrap required, opened a new log
I20260928 12:21:38.483309 7566 raft_consensus.cc:348] T 00000000000000000000000000000000 P de2f4df0533c4c0dbf020214e3af2a84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2f4df0533c4c0dbf020214e3af2a84" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 36119 } }
I20260928 12:21:38.483459 7566 raft_consensus.cc:374] T 00000000000000000000000000000000 P de2f4df0533c4c0dbf020214e3af2a84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:38.483481 7566 raft_consensus.cc:729] T 00000000000000000000000000000000 P de2f4df0533c4c0dbf020214e3af2a84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de2f4df0533c4c0dbf020214e3af2a84, State: Initialized, Role: FOLLOWER
I20260928 12:21:38.483639 7566 consensus_queue.cc:260] T 00000000000000000000000000000000 P de2f4df0533c4c0dbf020214e3af2a84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2f4df0533c4c0dbf020214e3af2a84" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 36119 } }
I20260928 12:21:38.483727 7566 raft_consensus.cc:388] T 00000000000000000000000000000000 P de2f4df0533c4c0dbf020214e3af2a84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:21:38.483758 7566 raft_consensus.cc:482] T 00000000000000000000000000000000 P de2f4df0533c4c0dbf020214e3af2a84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:21:38.483801 7566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de2f4df0533c4c0dbf020214e3af2a84 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:38.483829 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 7499
I20260928 12:21:38.483933 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:21:38.484120 2272 external_mini_cluster.cc:1428] Setting key 3a9bd03f2831b915ac8d8255a968914f
I20260928 12:21:38.484486 7566 raft_consensus.cc:504] T 00000000000000000000000000000000 P de2f4df0533c4c0dbf020214e3af2a84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2f4df0533c4c0dbf020214e3af2a84" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 36119 } }
I20260928 12:21:38.484592 7566 leader_election.cc:302] T 00000000000000000000000000000000 P de2f4df0533c4c0dbf020214e3af2a84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de2f4df0533c4c0dbf020214e3af2a84; no voters:
I20260928 12:21:38.484766 7566 leader_election.cc:288] T 00000000000000000000000000000000 P de2f4df0533c4c0dbf020214e3af2a84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:21:38.484843 7571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de2f4df0533c4c0dbf020214e3af2a84 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:21:38.484965 7571 raft_consensus.cc:686] T 00000000000000000000000000000000 P de2f4df0533c4c0dbf020214e3af2a84 [term 1 LEADER]: Becoming Leader. State: Replica: de2f4df0533c4c0dbf020214e3af2a84, State: Running, Role: LEADER
I20260928 12:21:38.485060 7566 sys_catalog.cc:564] T 00000000000000000000000000000000 P de2f4df0533c4c0dbf020214e3af2a84 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:21:38.485049 7571 consensus_queue.cc:237] T 00000000000000000000000000000000 P de2f4df0533c4c0dbf020214e3af2a84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2f4df0533c4c0dbf020214e3af2a84" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 36119 } }
I20260928 12:21:38.485512 7572 sys_catalog.cc:455] T 00000000000000000000000000000000 P de2f4df0533c4c0dbf020214e3af2a84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de2f4df0533c4c0dbf020214e3af2a84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2f4df0533c4c0dbf020214e3af2a84" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 36119 } } }
I20260928 12:21:38.485656 7572 sys_catalog.cc:458] T 00000000000000000000000000000000 P de2f4df0533c4c0dbf020214e3af2a84 [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:38.485641 7573 sys_catalog.cc:455] T 00000000000000000000000000000000 P de2f4df0533c4c0dbf020214e3af2a84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de2f4df0533c4c0dbf020214e3af2a84. Latest consensus state: current_term: 1 leader_uuid: "de2f4df0533c4c0dbf020214e3af2a84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2f4df0533c4c0dbf020214e3af2a84" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 36119 } } }
I20260928 12:21:38.485793 7573 sys_catalog.cc:458] T 00000000000000000000000000000000 P de2f4df0533c4c0dbf020214e3af2a84 [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:38.486035 7580 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:21:38.486622 7580 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:21:38.488386 7580 catalog_manager.cc:1347] Generated new cluster ID: b2092984efb8441bae5b479fbc686ed3
I20260928 12:21:38.488440 7580 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:21:38.490520 7570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:38.484631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33164 (local address 127.2.56.62:36119)
0928 12:21:38.484863 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:38.484874 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:38.484931 (+ 57us) server_negotiation.cc:407] Connection header received
0928 12:21:38.485162 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:38.485171 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:38.485251 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:38.485407 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:38.485832 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:38.487310 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:38.488403 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:38.488897 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:38.488967 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:38.489835 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:38.489840 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:38.489840 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:21:38.490027 (+ 187us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:38.490068 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:21:38.490080 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:38.490137 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:38.490208 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 12:21:38.490301 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":109,"threads_started":1}
I20260928 12:21:38.491397 2272 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:21:38.511363 7580 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:21:38.512147 7580 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:21:38.517743 7580 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de2f4df0533c4c0dbf020214e3af2a84: Generated new TSK 0
I20260928 12:21:38.517961 7580 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 12:21:38.550323 7596 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 12:21:38.550436 7596 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 12:21:38.550751 7570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:38.528704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33166 (local address 127.2.56.62:36119)
0928 12:21:38.528761 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:38.528763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:38.528875 (+ 112us) server_negotiation.cc:407] Connection header received
0928 12:21:38.528964 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:38.528964 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:38.528976 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:38.529036 (+ 60us) server_negotiation.cc:226] Negotiated authn=JWT
0928 12:21:38.529176 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:38.529767 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:38.530954 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:38.531152 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:38.531195 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:38.531371 (+ 176us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 12:21:38.550502 (+ 19131us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42909/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 12:21:38.550684 (+ 182us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33166: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42909/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":32}
W20260928 12:21:38.550823 7570 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33166: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42909/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 12:21:38.550896 7595 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:21:38.528648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:36119 (local address 127.0.0.1:33166)
0928 12:21:38.528799 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:38.528811 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:38.528892 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:38.529050 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:38.529051 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:38.529054 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0928 12:21:38.529122 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:38.529125 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:38.529844 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:38.529845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:38.530866 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:38.530870 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:38.531212 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:38.531216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:38.531283 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:38.531290 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 12:21:38.550648 (+ 19358us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42909/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 12:21:38.550786 (+ 138us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.56.62:36119: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42909/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
I20260928 12:21:38.551371 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 7499
2026-09-28T12:21:38Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (238 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 12:21:38.589695 2272 mini_oidc.cc:152] Starting JWKS server
I20260928 12:21:38.590428 2272 webserver.cc:460] Webserver started at https://127.0.0.1:43135/ using document root <none> and password file <none>
I20260928 12:21:38.590476 2272 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 12:21:38.590607 2272 webserver.cc:460] Webserver started at http://127.0.0.1:41953/ using document root <none> and password file <none>
2026-09-28T12:21:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:21:38Z Disabled control of system clock
I20260928 12:21:38.599081 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:43411
--webserver_interface=127.2.56.62
--webserver_port=0
--builtin_ntp_servers=127.2.56.20:43239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:43411
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43135/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 12:21:38.685971 7602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:38.686213 7602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:38.686242 7602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:38.687454 7602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:21:38.687495 7602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:38.687512 7602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:38.687527 7602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:21:38.687542 7602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 12:21:38.687559 7602 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 12:21:38.687580 7602 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43135/jwks.json
I20260928 12:21:38.688949 7602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:43239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:43411
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:43411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43135/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:38.689200 7602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:38.689517 7602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:38.694406 7607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:38.694399 7608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:38.694428 7602 server_base.cc:1034] running on GCE node
W20260928 12:21:38.694403 7610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:38.694904 7602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:38.695212 7602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:38.696349 7602 hybrid_clock.cc:648] HybridClock initialized: now 1790598098696336 us; error 31 us; skew 500 ppm
I20260928 12:21:38.697611 7602 webserver.cc:460] Webserver started at http://127.2.56.62:37421/ using document root <none> and password file <none>
I20260928 12:21:38.697839 7602 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:38.697888 7602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:38.698019 7602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:38.699146 7602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "9c2fdac6b5d847f295288c2be7591904"
format_stamp: "Formatted at 2026-09-28 12:21:38 on dist-test-slave-920p"
server_key: "62dd5a3966c3843dc267111573622baf"
server_key_iv: "105aa55ce0286fe4e715452b89fb8426"
server_key_version: "encryptionkey@0"
I20260928 12:21:38.699530 7602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "9c2fdac6b5d847f295288c2be7591904"
format_stamp: "Formatted at 2026-09-28 12:21:38 on dist-test-slave-920p"
server_key: "62dd5a3966c3843dc267111573622baf"
server_key_iv: "105aa55ce0286fe4e715452b89fb8426"
server_key_version: "encryptionkey@0"
I20260928 12:21:38.700834 7602 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:21:38.701535 7616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:38.701689 7602 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:21:38.701773 7602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "9c2fdac6b5d847f295288c2be7591904"
format_stamp: "Formatted at 2026-09-28 12:21:38 on dist-test-slave-920p"
server_key: "62dd5a3966c3843dc267111573622baf"
server_key_iv: "105aa55ce0286fe4e715452b89fb8426"
server_key_version: "encryptionkey@0"
I20260928 12:21:38.701834 7602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:38.716948 7602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:38.717329 7602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:38.717475 7602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:38.721737 7602 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:43411
I20260928 12:21:38.721776 7668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:43411 every 8 connection(s)
I20260928 12:21:38.722157 7602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:21:38.722915 7669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:38.723537 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 7602
I20260928 12:21:38.723618 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:21:38.723809 2272 external_mini_cluster.cc:1428] Setting key 48f770134ce9ae17e84d3b3f59480185
I20260928 12:21:38.726140 7669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c2fdac6b5d847f295288c2be7591904: Bootstrap starting.
I20260928 12:21:38.726824 7669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c2fdac6b5d847f295288c2be7591904: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:38.727159 7669 log.cc:824] T 00000000000000000000000000000000 P 9c2fdac6b5d847f295288c2be7591904: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:38.727854 7669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c2fdac6b5d847f295288c2be7591904: No bootstrap required, opened a new log
I20260928 12:21:38.729599 7672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:38.724353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37988 (local address 127.2.56.62:43411)
0928 12:21:38.724551 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:38.724560 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:38.724587 (+ 27us) server_negotiation.cc:407] Connection header received
0928 12:21:38.725088 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:38.725096 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:38.725164 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:38.725335 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:38.725550 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:38.726509 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:38.727571 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:38.728044 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:38.728143 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:38.728864 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:38.728867 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:38.728868 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:21:38.729048 (+ 180us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:38.729084 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:21:38.729096 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:38.729146 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:38.729234 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 12:21:38.729402 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":62,"threads_started":1}
I20260928 12:21:38.730011 7669 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c2fdac6b5d847f295288c2be7591904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2fdac6b5d847f295288c2be7591904" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43411 } }
I20260928 12:21:38.730172 7669 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c2fdac6b5d847f295288c2be7591904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:38.730214 7669 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c2fdac6b5d847f295288c2be7591904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c2fdac6b5d847f295288c2be7591904, State: Initialized, Role: FOLLOWER
I20260928 12:21:38.730341 7669 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c2fdac6b5d847f295288c2be7591904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2fdac6b5d847f295288c2be7591904" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43411 } }
I20260928 12:21:38.730424 7669 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c2fdac6b5d847f295288c2be7591904 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:21:38.730464 7669 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c2fdac6b5d847f295288c2be7591904 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:21:38.730515 7669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c2fdac6b5d847f295288c2be7591904 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:38.730692 2272 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:21:38.731145 7669 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c2fdac6b5d847f295288c2be7591904 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2fdac6b5d847f295288c2be7591904" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43411 } }
I20260928 12:21:38.731244 7669 leader_election.cc:302] T 00000000000000000000000000000000 P 9c2fdac6b5d847f295288c2be7591904 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c2fdac6b5d847f295288c2be7591904; no voters:
I20260928 12:21:38.731403 7669 leader_election.cc:288] T 00000000000000000000000000000000 P 9c2fdac6b5d847f295288c2be7591904 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:21:38.731509 7674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c2fdac6b5d847f295288c2be7591904 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:21:38.731653 7674 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c2fdac6b5d847f295288c2be7591904 [term 1 LEADER]: Becoming Leader. State: Replica: 9c2fdac6b5d847f295288c2be7591904, State: Running, Role: LEADER
I20260928 12:21:38.731678 7669 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c2fdac6b5d847f295288c2be7591904 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:21:38.731745 7674 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c2fdac6b5d847f295288c2be7591904 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2fdac6b5d847f295288c2be7591904" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43411 } }
I20260928 12:21:38.732100 7675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c2fdac6b5d847f295288c2be7591904 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c2fdac6b5d847f295288c2be7591904" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2fdac6b5d847f295288c2be7591904" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43411 } } }
I20260928 12:21:38.732204 7675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c2fdac6b5d847f295288c2be7591904 [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:38.732100 7676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c2fdac6b5d847f295288c2be7591904 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c2fdac6b5d847f295288c2be7591904. Latest consensus state: current_term: 1 leader_uuid: "9c2fdac6b5d847f295288c2be7591904" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2fdac6b5d847f295288c2be7591904" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43411 } } }
I20260928 12:21:38.732396 7676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c2fdac6b5d847f295288c2be7591904 [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:38.732493 7681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:21:38.733078 7681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:21:38.734722 7681 catalog_manager.cc:1347] Generated new cluster ID: 450f3e2328da48c3a6a365b49219b1c3
I20260928 12:21:38.734763 7681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:21:38.739240 7681 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:21:38.739750 7681 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:21:38.744807 7681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c2fdac6b5d847f295288c2be7591904: Generated new TSK 0
I20260928 12:21:38.744976 7681 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:21:38.759011 7672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:38.755488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37996 (local address 127.2.56.62:43411)
0928 12:21:38.755546 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:38.755548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:38.755661 (+ 113us) server_negotiation.cc:407] Connection header received
0928 12:21:38.755744 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:38.755745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:38.755760 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:38.755825 (+ 65us) server_negotiation.cc:226] Negotiated authn=JWT
0928 12:21:38.755962 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:38.756498 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:38.757336 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:38.757547 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:38.757601 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:38.757781 (+ 180us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 12:21:38.758765 (+ 984us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43135/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 12:21:38.758911 (+ 146us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37996: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43135/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 12:21:38.759086 7672 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37996: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43135/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 12:21:38.760973 7698 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:21:38.755435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43411 (local address 127.0.0.1:37996)
0928 12:21:38.755591 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:38.755603 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:38.755674 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:38.755827 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:38.755828 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:38.755830 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 12:21:38.755899 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:38.755902 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:38.756589 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:38.756590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:38.757232 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:38.757234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:38.757603 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:38.757612 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:38.757674 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:38.757680 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 12:21:38.758875 (+ 1195us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43135/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 12:21:38.760864 (+ 1989us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.56.62:43411: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43135/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":1749,"thread_start_us":102,"threads_started":1}
I20260928 12:21:38.761168 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 7602
E20260928 12:21:38.773612 7699 webserver.cc:533] Webserver: sslize failed:
E20260928 12:21:38.773720 7699 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T12:21:38Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (209 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T12:21:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:21:38Z Disabled control of system clock
I20260928 12:21:38.806672 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:46377
--webserver_interface=127.2.56.62
--webserver_port=0
--builtin_ntp_servers=127.2.56.20:43715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:46377
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598072638673-2272-0/testchainca.cert with env {}
W20260928 12:21:38.894062 7703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:38.894308 7703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:38.894331 7703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:38.895581 7703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:21:38.895644 7703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:38.895656 7703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:38.895669 7703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:21:38.895681 7703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 12:21:38.895696 7703 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 12:21:38.895712 7703 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 12:21:38.898175 7703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:43715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:46377
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:46377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598072638673-2272-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:38.898700 7703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:38.898937 7703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:38.903916 7711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:38.903996 7708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:38.903916 7709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:38.904173 7703 server_base.cc:1034] running on GCE node
I20260928 12:21:38.904421 7703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:38.904731 7703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:38.905865 7703 hybrid_clock.cc:648] HybridClock initialized: now 1790598098905856 us; error 34 us; skew 500 ppm
I20260928 12:21:38.907178 7703 webserver.cc:460] Webserver started at http://127.2.56.62:42791/ using document root <none> and password file <none>
I20260928 12:21:38.907428 7703 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:38.907476 7703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:38.907608 7703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:38.908771 7703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "b36b2296c2254f778c7ef0040fa7095c"
format_stamp: "Formatted at 2026-09-28 12:21:38 on dist-test-slave-920p"
server_key: "c49fdf86cfcd27ac1dbfc4f356dbf8f3"
server_key_iv: "00a30564883cca85be65d08eb912c480"
server_key_version: "encryptionkey@0"
I20260928 12:21:38.909128 7703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "b36b2296c2254f778c7ef0040fa7095c"
format_stamp: "Formatted at 2026-09-28 12:21:38 on dist-test-slave-920p"
server_key: "c49fdf86cfcd27ac1dbfc4f356dbf8f3"
server_key_iv: "00a30564883cca85be65d08eb912c480"
server_key_version: "encryptionkey@0"
I20260928 12:21:38.910444 7703 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:21:38.911131 7717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:38.911288 7703 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:21:38.911386 7703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "b36b2296c2254f778c7ef0040fa7095c"
format_stamp: "Formatted at 2026-09-28 12:21:38 on dist-test-slave-920p"
server_key: "c49fdf86cfcd27ac1dbfc4f356dbf8f3"
server_key_iv: "00a30564883cca85be65d08eb912c480"
server_key_version: "encryptionkey@0"
I20260928 12:21:38.911448 7703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:38.919168 7703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:38.919528 7703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:38.919655 7703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:38.923822 7703 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:46377
I20260928 12:21:38.923856 7769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:46377 every 8 connection(s)
I20260928 12:21:38.924188 7703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:21:38.925190 7770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:38.927762 7770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b36b2296c2254f778c7ef0040fa7095c: Bootstrap starting.
I20260928 12:21:38.928385 7770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b36b2296c2254f778c7ef0040fa7095c: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:38.928664 7770 log.cc:824] T 00000000000000000000000000000000 P b36b2296c2254f778c7ef0040fa7095c: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:38.929337 7770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b36b2296c2254f778c7ef0040fa7095c: No bootstrap required, opened a new log
I20260928 12:21:38.931046 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 7703
I20260928 12:21:38.931139 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:21:38.931031 7770 raft_consensus.cc:348] T 00000000000000000000000000000000 P b36b2296c2254f778c7ef0040fa7095c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b36b2296c2254f778c7ef0040fa7095c" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 46377 } }
I20260928 12:21:38.931200 7770 raft_consensus.cc:374] T 00000000000000000000000000000000 P b36b2296c2254f778c7ef0040fa7095c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:38.931243 7770 raft_consensus.cc:729] T 00000000000000000000000000000000 P b36b2296c2254f778c7ef0040fa7095c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36b2296c2254f778c7ef0040fa7095c, State: Initialized, Role: FOLLOWER
I20260928 12:21:38.931329 2272 external_mini_cluster.cc:1428] Setting key eeb5f5ace5e70d863795eed97cf1d2d9
I20260928 12:21:38.931399 7770 consensus_queue.cc:260] T 00000000000000000000000000000000 P b36b2296c2254f778c7ef0040fa7095c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b36b2296c2254f778c7ef0040fa7095c" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 46377 } }
I20260928 12:21:38.931483 7770 raft_consensus.cc:388] T 00000000000000000000000000000000 P b36b2296c2254f778c7ef0040fa7095c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:21:38.931514 7770 raft_consensus.cc:482] T 00000000000000000000000000000000 P b36b2296c2254f778c7ef0040fa7095c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:21:38.931563 7770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b36b2296c2254f778c7ef0040fa7095c [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:38.932194 7770 raft_consensus.cc:504] T 00000000000000000000000000000000 P b36b2296c2254f778c7ef0040fa7095c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b36b2296c2254f778c7ef0040fa7095c" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 46377 } }
I20260928 12:21:38.932294 7770 leader_election.cc:302] T 00000000000000000000000000000000 P b36b2296c2254f778c7ef0040fa7095c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b36b2296c2254f778c7ef0040fa7095c; no voters:
I20260928 12:21:38.932454 7770 leader_election.cc:288] T 00000000000000000000000000000000 P b36b2296c2254f778c7ef0040fa7095c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:21:38.932482 7775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b36b2296c2254f778c7ef0040fa7095c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:21:38.932632 7775 raft_consensus.cc:686] T 00000000000000000000000000000000 P b36b2296c2254f778c7ef0040fa7095c [term 1 LEADER]: Becoming Leader. State: Replica: b36b2296c2254f778c7ef0040fa7095c, State: Running, Role: LEADER
I20260928 12:21:38.932777 7770 sys_catalog.cc:564] T 00000000000000000000000000000000 P b36b2296c2254f778c7ef0040fa7095c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:21:38.932760 7775 consensus_queue.cc:237] T 00000000000000000000000000000000 P b36b2296c2254f778c7ef0040fa7095c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b36b2296c2254f778c7ef0040fa7095c" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 46377 } }
I20260928 12:21:38.933210 7777 sys_catalog.cc:455] T 00000000000000000000000000000000 P b36b2296c2254f778c7ef0040fa7095c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b36b2296c2254f778c7ef0040fa7095c. Latest consensus state: current_term: 1 leader_uuid: "b36b2296c2254f778c7ef0040fa7095c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b36b2296c2254f778c7ef0040fa7095c" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 46377 } } }
I20260928 12:21:38.933323 7777 sys_catalog.cc:458] T 00000000000000000000000000000000 P b36b2296c2254f778c7ef0040fa7095c [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:38.933530 7776 sys_catalog.cc:455] T 00000000000000000000000000000000 P b36b2296c2254f778c7ef0040fa7095c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b36b2296c2254f778c7ef0040fa7095c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b36b2296c2254f778c7ef0040fa7095c" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 46377 } } }
I20260928 12:21:38.933595 7776 sys_catalog.cc:458] T 00000000000000000000000000000000 P b36b2296c2254f778c7ef0040fa7095c [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:38.934051 7783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:21:38.934612 7783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:21:38.936299 7783 catalog_manager.cc:1347] Generated new cluster ID: c8dba5b7211241fa94392007fbde75fd
I20260928 12:21:38.936350 7783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:21:38.937469 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:38.931813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41740 (local address 127.2.56.62:46377)
0928 12:21:38.932030 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:38.932040 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:38.932073 (+ 33us) server_negotiation.cc:407] Connection header received
0928 12:21:38.932186 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:38.932193 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:38.932264 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:38.932422 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:38.932656 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:38.934383 (+ 1727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:38.935264 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:38.935898 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:38.936015 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:38.936731 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:38.936735 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:38.936735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:21:38.936943 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:38.936979 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:21:38.936991 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:38.937044 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:38.937122 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 12:21:38.937205 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":80,"threads_started":1}
I20260928 12:21:38.942237 2272 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:21:38.945152 7783 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:21:38.945772 7783 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:21:38.948779 7783 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b36b2296c2254f778c7ef0040fa7095c: Generated new TSK 0
I20260928 12:21:38.948969 7783 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:21:39.443929 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:38.956953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41746 (local address 127.2.56.62:46377)
0928 12:21:38.957002 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:38.957004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:38.957068 (+ 64us) server_negotiation.cc:407] Connection header received
0928 12:21:38.957159 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:38.957160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:38.957171 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:38.957231 (+ 60us) server_negotiation.cc:226] Negotiated authn=JWT
0928 12:21:38.957396 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:38.957878 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:38.959048 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:38.959247 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:38.959299 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:38.959476 (+ 177us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 12:21:39.443636 (+484160us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 12:21:39.443835 (+ 199us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41746: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":25}
W20260928 12:21:39.444017 7774 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41746: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 12:21:39.444062 7799 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:21:38.956859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:46377 (local address 127.0.0.1:41746)
0928 12:21:38.957003 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:38.957015 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:38.957091 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:38.957245 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:38.957246 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:38.957248 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 12:21:38.957338 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:38.957342 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:38.957968 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:38.957969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:38.958970 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:38.958973 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:38.959314 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:38.959320 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:38.959379 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:38.959385 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 12:21:39.443807 (+484422us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 12:21:39.443958 (+ 151us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.56.62:46377: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
I20260928 12:21:39.444389 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 7703
2026-09-28T12:21:39Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (663 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (95 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (96 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:21:39 dist-test-slave-920p krb5kdc[7811](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:21:39 dist-test-slave-920p krb5kdc[7811](info): set up 2 sockets
Sep 28 12:21:39 dist-test-slave-920p krb5kdc[7811](info): commencing operation
krb5kdc: starting...
W20260928 12:21:40.601783 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:21:40 dist-test-slave-920p krb5kdc[7811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:21:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:21:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
I20260928 12:21:40.722582 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:41719
--webserver_interface=127.2.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:42339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:41719
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:40.811132 7824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:40.811376 7824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:40.811399 7824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:40.812577 7824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:21:40.812613 7824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:40.812626 7824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:40.812637 7824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:21:40.812647 7824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:21:40.814200 7824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:42339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:41719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:41719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:40.814426 7824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:40.814627 7824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:40.819540 7829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:40.819624 7832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:40.819605 7830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:40.819850 7824 server_base.cc:1034] running on GCE node
I20260928 12:21:40.820125 7824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:40.820427 7824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:40.821568 7824 hybrid_clock.cc:648] HybridClock initialized: now 1790598100821550 us; error 30 us; skew 500 ppm
Sep 28 12:21:40 dist-test-slave-920p krb5kdc[7811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598100, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:40.824182 7824 init.cc:377] Logged in from keytab as kudu/127.2.56.62@KRBTEST.COM (short username kudu)
I20260928 12:21:40.824640 7824 webserver.cc:460] Webserver started at http://127.2.56.62:39117/ using document root <none> and password file <none>
I20260928 12:21:40.824880 7824 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:40.824935 7824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:40.825073 7824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:40.826290 7824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "0201356a71ee4f468eb030581b651faa"
format_stamp: "Formatted at 2026-09-28 12:21:40 on dist-test-slave-920p"
server_key: "c4b99592a0d870f44a246f61a539af27"
server_key_iv: "af2cd3f3478c058c3f2f49511565b015"
server_key_version: "encryptionkey@0"
I20260928 12:21:40.826692 7824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "0201356a71ee4f468eb030581b651faa"
format_stamp: "Formatted at 2026-09-28 12:21:40 on dist-test-slave-920p"
server_key: "c4b99592a0d870f44a246f61a539af27"
server_key_iv: "af2cd3f3478c058c3f2f49511565b015"
server_key_version: "encryptionkey@0"
I20260928 12:21:40.828079 7824 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:21:40.828825 7839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:40.828989 7824 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:21:40.829084 7824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "0201356a71ee4f468eb030581b651faa"
format_stamp: "Formatted at 2026-09-28 12:21:40 on dist-test-slave-920p"
server_key: "c4b99592a0d870f44a246f61a539af27"
server_key_iv: "af2cd3f3478c058c3f2f49511565b015"
server_key_version: "encryptionkey@0"
I20260928 12:21:40.829141 7824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:40.840969 7824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:40.843031 7824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:40.843166 7824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:40.847457 7824 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:41719
I20260928 12:21:40.847502 7891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:41719 every 8 connection(s)
I20260928 12:21:40.847863 7824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:21:40.848838 7892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:40.851536 7892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0201356a71ee4f468eb030581b651faa: Bootstrap starting.
I20260928 12:21:40.852200 7892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0201356a71ee4f468eb030581b651faa: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:40.852500 7892 log.cc:824] T 00000000000000000000000000000000 P 0201356a71ee4f468eb030581b651faa: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:40.853226 7892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0201356a71ee4f468eb030581b651faa: No bootstrap required, opened a new log
I20260928 12:21:40.855065 7892 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0201356a71ee4f468eb030581b651faa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0201356a71ee4f468eb030581b651faa" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 41719 } }
I20260928 12:21:40.855228 7892 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0201356a71ee4f468eb030581b651faa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:40.855262 7892 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0201356a71ee4f468eb030581b651faa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0201356a71ee4f468eb030581b651faa, State: Initialized, Role: FOLLOWER
I20260928 12:21:40.855412 7892 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0201356a71ee4f468eb030581b651faa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0201356a71ee4f468eb030581b651faa" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 41719 } }
I20260928 12:21:40.855496 7892 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0201356a71ee4f468eb030581b651faa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:21:40.855538 7892 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0201356a71ee4f468eb030581b651faa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:21:40.855590 7892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0201356a71ee4f468eb030581b651faa [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:40.856143 7892 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0201356a71ee4f468eb030581b651faa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0201356a71ee4f468eb030581b651faa" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 41719 } }
I20260928 12:21:40.856244 7892 leader_election.cc:302] T 00000000000000000000000000000000 P 0201356a71ee4f468eb030581b651faa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0201356a71ee4f468eb030581b651faa; no voters:
I20260928 12:21:40.856422 7892 leader_election.cc:288] T 00000000000000000000000000000000 P 0201356a71ee4f468eb030581b651faa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:21:40.856488 7895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0201356a71ee4f468eb030581b651faa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:21:40.856690 7892 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0201356a71ee4f468eb030581b651faa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:21:40.856844 7895 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0201356a71ee4f468eb030581b651faa [term 1 LEADER]: Becoming Leader. State: Replica: 0201356a71ee4f468eb030581b651faa, State: Running, Role: LEADER
I20260928 12:21:40.856946 7895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0201356a71ee4f468eb030581b651faa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0201356a71ee4f468eb030581b651faa" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 41719 } }
I20260928 12:21:40.857388 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 7824
I20260928 12:21:40.857383 7897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0201356a71ee4f468eb030581b651faa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0201356a71ee4f468eb030581b651faa. Latest consensus state: current_term: 1 leader_uuid: "0201356a71ee4f468eb030581b651faa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0201356a71ee4f468eb030581b651faa" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 41719 } } }
I20260928 12:21:40.857461 7897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0201356a71ee4f468eb030581b651faa [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:40.857506 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:21:40.857610 7896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0201356a71ee4f468eb030581b651faa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0201356a71ee4f468eb030581b651faa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0201356a71ee4f468eb030581b651faa" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 41719 } } }
I20260928 12:21:40.857656 7896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0201356a71ee4f468eb030581b651faa [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:40.857712 2272 external_mini_cluster.cc:1428] Setting key ee93bfb88af25ade600e454b8f13850d
I20260928 12:21:40.858234 7904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:21:40.858834 7904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:21:40.860569 7904 catalog_manager.cc:1347] Generated new cluster ID: baf57b0c1e244e95b3a93644117a068f
I20260928 12:21:40.860620 7904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:21:40 dist-test-slave-920p krb5kdc[7811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:40.868556 7904 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:21:40.868877 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:40.858242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48860 (local address 127.2.56.62:41719)
0928 12:21:40.858513 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:40.858524 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:40.858558 (+ 34us) server_negotiation.cc:407] Connection header received
0928 12:21:40.858678 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:40.858688 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:40.858755 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:40.858924 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:40.859717 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:40.860982 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:40.861805 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:40.862266 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:40.862364 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:40.864801 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:40.864806 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:40.864806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:40.864821 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:40.867132 (+ 2311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:40.867498 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:40.867507 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:40.867508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:40.867615 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:40.867799 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:40.867799 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:40.867800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:40.868096 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:40.868329 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:40.868407 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 12:21:40.868563 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":73,"threads_started":1}
I20260928 12:21:40.869750 7904 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:21:40.878324 7904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0201356a71ee4f468eb030581b651faa: Generated new TSK 0
I20260928 12:21:40.878556 7904 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab.
Entry for principal kudu/127.2.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab.
I20260928 12:21:40.911850 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.1:0
--local_ip_for_outbound_sockets=127.2.56.1
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:41719
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:42339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:41.018149 7918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:41.018453 7918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:41.018496 7918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:41.020287 7918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:41.020344 7918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:41.020382 7918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.1
I20260928 12:21:41.022594 7918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:42339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:41719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.1
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:41.022926 7918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:41.023244 7918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:41.030109 7924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:41.030128 7923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:41.030463 7926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:41.030836 7918 server_base.cc:1034] running on GCE node
I20260928 12:21:41.031064 7918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:41.031344 7918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:41.032485 7918 hybrid_clock.cc:648] HybridClock initialized: now 1790598101032470 us; error 34 us; skew 500 ppm
Sep 28 12:21:41 dist-test-slave-920p krb5kdc[7811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598101, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:41.035288 7918 init.cc:377] Logged in from keytab as kudu/127.2.56.1@KRBTEST.COM (short username kudu)
I20260928 12:21:41.035720 7918 webserver.cc:460] Webserver started at http://127.2.56.1:46455/ using document root <none> and password file <none>
I20260928 12:21:41.035938 7918 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:41.035980 7918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:41.036082 7918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:41.037222 7918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-0/data/instance:
uuid: "50c9b1e8e1454b59b47238126d8c4619"
format_stamp: "Formatted at 2026-09-28 12:21:41 on dist-test-slave-920p"
server_key: "73c1f70ab69154e036c67bed6b92d7f6"
server_key_iv: "5ded0b0d4b90ec41c17a3f2b5d5763f0"
server_key_version: "encryptionkey@0"
I20260928 12:21:41.037581 7918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance:
uuid: "50c9b1e8e1454b59b47238126d8c4619"
format_stamp: "Formatted at 2026-09-28 12:21:41 on dist-test-slave-920p"
server_key: "73c1f70ab69154e036c67bed6b92d7f6"
server_key_iv: "5ded0b0d4b90ec41c17a3f2b5d5763f0"
server_key_version: "encryptionkey@0"
I20260928 12:21:41.038815 7918 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:21:41.039476 7933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:41.039636 7918 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:41.039722 7918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-0/wal
uuid: "50c9b1e8e1454b59b47238126d8c4619"
format_stamp: "Formatted at 2026-09-28 12:21:41 on dist-test-slave-920p"
server_key: "73c1f70ab69154e036c67bed6b92d7f6"
server_key_iv: "5ded0b0d4b90ec41c17a3f2b5d5763f0"
server_key_version: "encryptionkey@0"
I20260928 12:21:41.039796 7918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:41.073994 7918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:41.076027 7918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:41.076159 7918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:41.076392 7918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:21:41.076701 7918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:41.076731 7918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:41.076756 7918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:41.076766 7918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:41.083015 7918 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.1:33143
I20260928 12:21:41.083086 8046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.1:33143 every 8 connection(s)
I20260928 12:21:41.083424 7918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
Sep 28 12:21:41 dist-test-slave-920p krb5kdc[7811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598101, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:41.091938 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 7918
I20260928 12:21:41.092069 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance
I20260928 12:21:41.092316 2272 external_mini_cluster.cc:1428] Setting key 59ebdd209cbb7eca1cec51c741b8fddc
I20260928 12:21:41.094847 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:41.084171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:57793 (local address 127.2.56.62:41719)
0928 12:21:41.084225 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:41.084227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:41.084441 (+ 214us) server_negotiation.cc:407] Connection header received
0928 12:21:41.085553 (+ 1112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:41.085555 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:41.085575 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:41.085635 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:41.086373 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:41.086960 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:41.088045 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:41.088279 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:41.088340 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:41.091294 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:41.091297 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:41.091297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:41.091307 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:41.093053 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:41.093612 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:41.093614 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:41.093614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:41.093717 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:41.094071 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:41.094072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:41.094073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:41.094236 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:41.094301 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:41.094607 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 12:21:41.094734 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:21:41.095212 8049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:41.084139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:41719 (local address 127.2.56.1:57793)
0928 12:21:41.084300 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:41.084343 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:41.085413 (+ 1070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:41.085734 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:41.085739 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:41.086038 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:41.086272 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:41.086278 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:41.087050 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:41.087051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:41.087957 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:41.087959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:41.088357 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:41.088358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:41.088523 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:41.089326 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:41.089340 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:41.091153 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:41.093141 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:41.093156 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:41.093160 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:41.093532 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:41.093793 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:41.093794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:41.093794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:41.093995 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:41.094289 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:41.094294 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:41.094474 (+ 180us) client_negotiation.cc:769] Sending connection context
0928 12:21:41.094567 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 12:21:41.094745 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20260928 12:21:41.095815 8047 heartbeater.cc:347] Connected to a master server at 127.2.56.62:41719
I20260928 12:21:41.095956 8047 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:41.096190 8047 heartbeater.cc:511] Master 127.2.56.62:41719 requested a full tablet report, sending...
I20260928 12:21:41.096844 7856 ts_manager.cc:194] Registered new tserver with Master: 50c9b1e8e1454b59b47238126d8c4619 (127.2.56.1:33143)
I20260928 12:21:41.097819 7856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.1@KRBTEST.COM'} at 127.2.56.1:57793
WARNING: no policy specified for kudu/127.2.56.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab.
Entry for principal kudu/127.2.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab.
I20260928 12:21:41.121057 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.2:0
--local_ip_for_outbound_sockets=127.2.56.2
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:41719
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:42339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:41.208587 8052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:41.208833 8052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:41.208868 8052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:41.210191 8052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:41.210249 8052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:41.210287 8052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.2
I20260928 12:21:41.211905 8052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:42339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:41719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.2
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:41.212183 8052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:41.212443 8052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:41.217857 8057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:41.217859 8060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:41.217890 8058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:41.218200 8052 server_base.cc:1034] running on GCE node
I20260928 12:21:41.218448 8052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:41.218711 8052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:41.219869 8052 hybrid_clock.cc:648] HybridClock initialized: now 1790598101219851 us; error 39 us; skew 500 ppm
Sep 28 12:21:41 dist-test-slave-920p krb5kdc[7811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598101, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:41.222663 8052 init.cc:377] Logged in from keytab as kudu/127.2.56.2@KRBTEST.COM (short username kudu)
I20260928 12:21:41.223166 8052 webserver.cc:460] Webserver started at http://127.2.56.2:43739/ using document root <none> and password file <none>
I20260928 12:21:41.223387 8052 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:41.223431 8052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:41.223537 8052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:41.224682 8052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-1/data/instance:
uuid: "259e156d8f674451a7932cf7e1b2cb50"
format_stamp: "Formatted at 2026-09-28 12:21:41 on dist-test-slave-920p"
server_key: "0642cdc4b02da4127b0204d8863700d2"
server_key_iv: "f8489adef3002441191b99c2274757a2"
server_key_version: "encryptionkey@0"
I20260928 12:21:41.225008 8052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance:
uuid: "259e156d8f674451a7932cf7e1b2cb50"
format_stamp: "Formatted at 2026-09-28 12:21:41 on dist-test-slave-920p"
server_key: "0642cdc4b02da4127b0204d8863700d2"
server_key_iv: "f8489adef3002441191b99c2274757a2"
server_key_version: "encryptionkey@0"
I20260928 12:21:41.226382 8052 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:21:41.227098 8067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:41.227257 8052 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:41.227339 8052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-1/wal
uuid: "259e156d8f674451a7932cf7e1b2cb50"
format_stamp: "Formatted at 2026-09-28 12:21:41 on dist-test-slave-920p"
server_key: "0642cdc4b02da4127b0204d8863700d2"
server_key_iv: "f8489adef3002441191b99c2274757a2"
server_key_version: "encryptionkey@0"
I20260928 12:21:41.227401 8052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:41.235419 8052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:41.237397 8052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:41.237521 8052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:41.237722 8052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:21:41.237998 8052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:41.238026 8052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:41.238052 8052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:41.238068 8052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:41.244385 8052 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.2:46515
I20260928 12:21:41.244452 8180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.2:46515 every 8 connection(s)
I20260928 12:21:41.244810 8052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
I20260928 12:21:41.246397 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 8052
I20260928 12:21:41.246511 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance
I20260928 12:21:41.246686 2272 external_mini_cluster.cc:1428] Setting key 2c68e7ee9a078e3851282ef2ac1d2af8
Sep 28 12:21:41 dist-test-slave-920p krb5kdc[7811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598101, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:41.255246 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:41.245622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:38329 (local address 127.2.56.62:41719)
0928 12:21:41.245714 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:41.245716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:41.245909 (+ 193us) server_negotiation.cc:407] Connection header received
0928 12:21:41.247062 (+ 1153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:41.247063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:41.247076 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:41.247131 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:41.247949 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:41.248668 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:41.249791 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:41.249995 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:41.250054 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:41.252417 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:41.252419 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:41.252427 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:41.252437 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:41.253930 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:41.254292 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:41.254294 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:41.254294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:41.254359 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:41.254575 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:41.254576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:41.254576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:41.254725 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:41.254795 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:41.255052 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 12:21:41.255143 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 12:21:41.255343 8183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:41.245596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:41719 (local address 127.2.56.2:38329)
0928 12:21:41.245776 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:41.245818 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:41.246961 (+ 1143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:41.247228 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:41.247233 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:41.247613 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:41.247846 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:41.247851 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:41.248745 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:41.248746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:41.249714 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:41.249717 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:41.250065 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:41.250066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:41.250203 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:41.250713 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:41.250725 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:41.252328 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:41.254002 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:41.254008 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:41.254010 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:41.254228 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:41.254428 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:41.254428 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:41.254429 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:41.254524 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:41.254781 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:41.254785 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:41.254915 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 12:21:41.255008 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 12:21:41.255143 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":97,"threads_started":1}
I20260928 12:21:41.255869 8181 heartbeater.cc:347] Connected to a master server at 127.2.56.62:41719
I20260928 12:21:41.255975 8181 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:41.256168 8181 heartbeater.cc:511] Master 127.2.56.62:41719 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.56.3@KRBTEST.COM; defaulting to no policy
I20260928 12:21:41.256588 7856 ts_manager.cc:194] Registered new tserver with Master: 259e156d8f674451a7932cf7e1b2cb50 (127.2.56.2:46515)
I20260928 12:21:41.257007 7856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.2@KRBTEST.COM'} at 127.2.56.2:38329
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab.
Entry for principal kudu/127.2.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab.
I20260928 12:21:41.272507 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.3:0
--local_ip_for_outbound_sockets=127.2.56.3
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:41719
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:42339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:41.358735 8186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:41.358960 8186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:41.358983 8186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:41.360121 8186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:41.360160 8186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:41.360186 8186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.3
I20260928 12:21:41.361670 8186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:42339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:41719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.3
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:41.361904 8186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:41.362138 8186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:41.367244 8191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:41.367244 8194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:41.367239 8192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:41.367499 8186 server_base.cc:1034] running on GCE node
I20260928 12:21:41.367705 8186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:41.367903 8186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:41.369035 8186 hybrid_clock.cc:648] HybridClock initialized: now 1790598101369020 us; error 30 us; skew 500 ppm
Sep 28 12:21:41 dist-test-slave-920p krb5kdc[7811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598101, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:41.371624 8186 init.cc:377] Logged in from keytab as kudu/127.2.56.3@KRBTEST.COM (short username kudu)
I20260928 12:21:41.372068 8186 webserver.cc:460] Webserver started at http://127.2.56.3:46299/ using document root <none> and password file <none>
I20260928 12:21:41.372258 8186 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:41.372296 8186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:41.372398 8186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:41.373531 8186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-2/data/instance:
uuid: "ecc4bd4771064fe7b083e20fde67c786"
format_stamp: "Formatted at 2026-09-28 12:21:41 on dist-test-slave-920p"
server_key: "01bbe62c39cd2944a383d9579349de5d"
server_key_iv: "5e0994efb89e84649c443c87ede345f3"
server_key_version: "encryptionkey@0"
I20260928 12:21:41.373852 8186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance:
uuid: "ecc4bd4771064fe7b083e20fde67c786"
format_stamp: "Formatted at 2026-09-28 12:21:41 on dist-test-slave-920p"
server_key: "01bbe62c39cd2944a383d9579349de5d"
server_key_iv: "5e0994efb89e84649c443c87ede345f3"
server_key_version: "encryptionkey@0"
I20260928 12:21:41.375051 8186 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:21:41.375672 8201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:41.375828 8186 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:41.375905 8186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-2/wal
uuid: "ecc4bd4771064fe7b083e20fde67c786"
format_stamp: "Formatted at 2026-09-28 12:21:41 on dist-test-slave-920p"
server_key: "01bbe62c39cd2944a383d9579349de5d"
server_key_iv: "5e0994efb89e84649c443c87ede345f3"
server_key_version: "encryptionkey@0"
I20260928 12:21:41.375963 8186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:41.389535 8186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:41.391273 8186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:41.391382 8186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:41.391584 8186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:21:41.391853 8186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:41.391881 8186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:41.391904 8186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:41.391919 8186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:41.398317 8186 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.3:44831
I20260928 12:21:41.398383 8314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.3:44831 every 8 connection(s)
I20260928 12:21:41.398710 8186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
Sep 28 12:21:41 dist-test-slave-920p krb5kdc[7811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598101, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:41.407537 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 8186
I20260928 12:21:41.407626 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance
I20260928 12:21:41.407829 2272 external_mini_cluster.cc:1428] Setting key 2b91cc0613e7036e89a9f37db963f477
I20260928 12:21:41.407974 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:41.399474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:37629 (local address 127.2.56.62:41719)
0928 12:21:41.399545 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:41.399547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:41.399746 (+ 199us) server_negotiation.cc:407] Connection header received
0928 12:21:41.400775 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:41.400776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:41.400786 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:41.400836 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:41.401583 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:41.402214 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:41.403237 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:41.403418 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:41.403464 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:41.405642 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:41.405643 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:41.405643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:41.405650 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:41.406772 (+ 1122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:41.407107 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:41.407108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:41.407109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:41.407153 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:41.407321 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:41.407322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:41.407322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:41.407415 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:41.407463 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:41.407794 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 12:21:41.407861 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 12:21:41.408208 8317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:41.399456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:41719 (local address 127.2.56.3:37629)
0928 12:21:41.399616 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:41.399656 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:41.400685 (+ 1029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:41.400908 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:41.400911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:41.401192 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:41.401479 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:41.401486 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:41.402288 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:41.402289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:41.403175 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:41.403178 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:41.403474 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:41.403475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:41.403580 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:41.404124 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:41.404134 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:41.405555 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:41.406833 (+ 1278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:41.406837 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:41.406839 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:41.407051 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:41.407193 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:41.407194 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:41.407194 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:41.407282 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:41.407485 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:41.407491 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:41.407665 (+ 174us) client_negotiation.cc:769] Sending connection context
0928 12:21:41.407779 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 12:21:41.407959 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
I20260928 12:21:41.408754 8315 heartbeater.cc:347] Connected to a master server at 127.2.56.62:41719
I20260928 12:21:41.408857 8315 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:41.409044 8315 heartbeater.cc:511] Master 127.2.56.62:41719 requested a full tablet report, sending...
I20260928 12:21:41.409417 7856 ts_manager.cc:194] Registered new tserver with Master: ecc4bd4771064fe7b083e20fde67c786 (127.2.56.3:44831)
I20260928 12:21:41.409788 7856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.3@KRBTEST.COM'} at 127.2.56.3:37629
I20260928 12:21:41.419561 2272 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/krb5kdc/test-admin.keytab.
Sep 28 12:21:41 dist-test-slave-920p krb5kdc[7811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:41.433928 2272 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 12:21:41.435662 8322 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:21:41.434865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47316 (local address 127.2.56.1:33143)
0928 12:21:41.435040 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:41.435046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:41.435057 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:21:41.435089 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:41.435091 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:41.435107 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:41.435182 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:41.435565 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47316: BlockingRecv error: recv got EOF from 127.0.0.1:47316 (error 108)
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":76,"threads_started":1}
W20260928 12:21:41.435756 2272 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/krb5kdc/krb5cc)
Sep 28 12:21:41 dist-test-slave-920p krb5kdc[7811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:41.436767 2272 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 12:21:41 dist-test-slave-920p krb5kdc[7811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:41.437664 2272 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 12:21:41 dist-test-slave-920p krb5kdc[7811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.1@KRBTEST.COM
I20260928 12:21:41.445151 8322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:41.438213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47318 (local address 127.2.56.1:33143)
0928 12:21:41.438265 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:41.438267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:41.438300 (+ 33us) server_negotiation.cc:407] Connection header received
0928 12:21:41.438354 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:41.438355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:41.438366 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:41.438424 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:41.438762 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:41.439553 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:41.440310 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:41.440669 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:41.440719 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:41.442785 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:41.442795 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:41.442795 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:41.442802 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:41.444215 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:41.444518 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:41.444521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:41.444521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:41.444569 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:41.444693 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:41.444694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:41.444694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:41.444823 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:41.444946 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:41.444979 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 12:21:41.445038 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260928 12:21:41.447325 8322 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:21:41.446704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47334 (local address 127.2.56.1:33143)
0928 12:21:41.446758 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:41.446761 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:41.446798 (+ 37us) server_negotiation.cc:407] Connection header received
0928 12:21:41.446852 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:41.446853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:41.446864 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:41.446905 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:41.447275 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47334: BlockingRecv error: recv got EOF from 127.0.0.1:47334 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260928 12:21:41.447372 2272 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/krb5kdc/krb5cc' of type FILE
Sep 28 12:21:41 dist-test-slave-920p krb5kdc[7811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:41.448239 2272 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 12:21:41 dist-test-slave-920p krb5kdc[7811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:41.448983 2272 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 12:21:41 dist-test-slave-920p krb5kdc[7811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.1@KRBTEST.COM
I20260928 12:21:41.455150 8322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:41.449507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47344 (local address 127.2.56.1:33143)
0928 12:21:41.449560 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:41.449562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:41.449604 (+ 42us) server_negotiation.cc:407] Connection header received
0928 12:21:41.449635 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:41.449635 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:41.449645 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:41.449686 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:41.450022 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:41.450549 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:41.451204 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:41.451417 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:41.451468 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:41.453296 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:41.453297 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:41.453297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:41.453304 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:41.454349 (+ 1045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:41.454620 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:41.454621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:41.454621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:41.454664 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:41.454786 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:41.454787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:41.454787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:41.454869 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:41.454935 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:41.455004 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 12:21:41.455063 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260928 12:21:41.456605 8322 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:21:41.456152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47356 (local address 127.2.56.1:33143)
0928 12:21:41.456188 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:41.456189 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:41.456240 (+ 51us) server_negotiation.cc:407] Connection header received
0928 12:21:41.456289 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:41.456289 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:41.456298 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:41.456335 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:41.456558 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47356: BlockingRecv error: recv got EOF from 127.0.0.1:47356 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 12:21:41.456710 2272 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598072638673-2272-0/krb5kdc/krb5cc' of type FILE
Sep 28 12:21:41 dist-test-slave-920p krb5kdc[7811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:41.457707 2272 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 12:21:41 dist-test-slave-920p krb5kdc[7811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:41.458495 2272 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 12:21:41 dist-test-slave-920p krb5kdc[7811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.1@KRBTEST.COM
I20260928 12:21:41.464533 8322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:41.459061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47368 (local address 127.2.56.1:33143)
0928 12:21:41.459100 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:41.459101 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:41.459149 (+ 48us) server_negotiation.cc:407] Connection header received
0928 12:21:41.459194 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:41.459194 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:41.459203 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:41.459235 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:41.459532 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:41.460024 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:41.460636 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:41.460810 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:41.460867 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:41.462659 (+ 1792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:41.462661 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:41.462661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:41.462667 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:41.463735 (+ 1068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:41.464011 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:41.464011 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:41.464012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:41.464054 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:41.464171 (+ 117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:41.464171 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:41.464172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:41.464252 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:41.464330 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:41.464394 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 12:21:41.464447 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:21:41.465219 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 7918
I20260928 12:21:41.470708 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 8052
I20260928 12:21:41.474802 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 8186
I20260928 12:21:41.478566 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 7824
2026-09-28T12:21:41Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1842 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:21:41 dist-test-slave-920p krb5kdc[8335](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:21:41 dist-test-slave-920p krb5kdc[8335](info): set up 2 sockets
Sep 28 12:21:41 dist-test-slave-920p krb5kdc[8335](info): commencing operation
krb5kdc: starting...
W20260928 12:21:42.433794 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:21:42 dist-test-slave-920p krb5kdc[8335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:21:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:21:42Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.62.keytab.
Entry for principal oryx/127.2.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.62.keytab.
I20260928 12:21:42.546309 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:43477
--webserver_interface=127.2.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.62.keytab
--principal=oryx/127.2.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:43591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:43477
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:42.634830 8348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:42.635090 8348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:42.635123 8348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:42.636355 8348 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:21:42.636397 8348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:42.636416 8348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:42.636431 8348 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:21:42.636446 8348 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:21:42.637985 8348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:43591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:43477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.62.keytab
--principal=oryx/127.2.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:43477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:42.638249 8348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:42.638494 8348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:42.643633 8353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:42.643637 8356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:42.643642 8354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:42.643950 8348 server_base.cc:1034] running on GCE node
I20260928 12:21:42.644200 8348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:42.644505 8348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:42.645656 8348 hybrid_clock.cc:648] HybridClock initialized: now 1790598102645635 us; error 32 us; skew 500 ppm
Sep 28 12:21:42 dist-test-slave-920p krb5kdc[8335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598102, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:42.652473 8348 init.cc:377] Logged in from keytab as oryx/127.2.56.62@KRBTEST.COM (short username oryx)
I20260928 12:21:42.652992 8348 webserver.cc:460] Webserver started at http://127.2.56.62:33493/ using document root <none> and password file <none>
I20260928 12:21:42.653214 8348 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:42.653285 8348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:42.653426 8348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:42.654606 8348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "262a43ba51f54d5ba1fc806531019e58"
format_stamp: "Formatted at 2026-09-28 12:21:42 on dist-test-slave-920p"
server_key: "fbd4659a2e49742465041b82fbd9047f"
server_key_iv: "92b96804447ee1467b8352f67774a491"
server_key_version: "encryptionkey@0"
I20260928 12:21:42.654974 8348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "262a43ba51f54d5ba1fc806531019e58"
format_stamp: "Formatted at 2026-09-28 12:21:42 on dist-test-slave-920p"
server_key: "fbd4659a2e49742465041b82fbd9047f"
server_key_iv: "92b96804447ee1467b8352f67774a491"
server_key_version: "encryptionkey@0"
I20260928 12:21:42.656307 8348 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:21:42.657173 8363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:42.657368 8348 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:21:42.657497 8348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "262a43ba51f54d5ba1fc806531019e58"
format_stamp: "Formatted at 2026-09-28 12:21:42 on dist-test-slave-920p"
server_key: "fbd4659a2e49742465041b82fbd9047f"
server_key_iv: "92b96804447ee1467b8352f67774a491"
server_key_version: "encryptionkey@0"
I20260928 12:21:42.657572 8348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:42.672338 8348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:42.674811 8348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:42.674963 8348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:42.679319 8348 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:43477
I20260928 12:21:42.679364 8415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:43477 every 8 connection(s)
I20260928 12:21:42.679731 8348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:21:42.680832 8416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:42.681365 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 8348
I20260928 12:21:42.681476 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:21:42.681653 2272 external_mini_cluster.cc:1428] Setting key d1fe4fb004635e0e4f2e31a8d1f32e55
I20260928 12:21:42.683995 8416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58: Bootstrap starting.
I20260928 12:21:42.684626 8416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:42.684908 8416 log.cc:824] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:42.685655 8416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58: No bootstrap required, opened a new log
I20260928 12:21:42.687973 8416 raft_consensus.cc:348] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } }
I20260928 12:21:42.688161 8416 raft_consensus.cc:374] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:42.688189 8416 raft_consensus.cc:729] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 262a43ba51f54d5ba1fc806531019e58, State: Initialized, Role: FOLLOWER
I20260928 12:21:42.688315 8416 consensus_queue.cc:260] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } }
I20260928 12:21:42.688405 8416 raft_consensus.cc:388] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:21:42.688448 8416 raft_consensus.cc:482] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:21:42.688498 8416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [term 0 FOLLOWER]: Advancing to term 1
Sep 28 12:21:42 dist-test-slave-920p krb5kdc[8335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM
I20260928 12:21:42.689085 8416 raft_consensus.cc:504] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } }
I20260928 12:21:42.689190 8416 leader_election.cc:302] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 262a43ba51f54d5ba1fc806531019e58; no voters:
I20260928 12:21:42.689388 8416 leader_election.cc:288] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:21:42.689430 8421 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:21:42.689549 8421 raft_consensus.cc:686] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [term 1 LEADER]: Becoming Leader. State: Replica: 262a43ba51f54d5ba1fc806531019e58, State: Running, Role: LEADER
I20260928 12:21:42.689646 8421 consensus_queue.cc:237] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } }
I20260928 12:21:42.689689 8416 sys_catalog.cc:564] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:21:42.690057 8422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "262a43ba51f54d5ba1fc806531019e58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } }
I20260928 12:21:42.690138 8422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:42.690289 8423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 262a43ba51f54d5ba1fc806531019e58. Latest consensus state: current_term: 1 leader_uuid: "262a43ba51f54d5ba1fc806531019e58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } }
I20260928 12:21:42.690353 8423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:42.690469 8427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:21:42.691061 8427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:21:42.692747 8427 catalog_manager.cc:1347] Generated new cluster ID: f5f2d04c9c0b48a1b4ff09425dfdc0bd
I20260928 12:21:42.692799 8427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:21:42.693570 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:42.682226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51686 (local address 127.2.56.62:43477)
0928 12:21:42.682401 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:42.682410 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:42.682440 (+ 30us) server_negotiation.cc:407] Connection header received
0928 12:21:42.682890 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:42.682898 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:42.682953 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:42.683081 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:42.683839 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:42.684884 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:42.685923 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:42.686350 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:42.686474 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:42.689160 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:42.689165 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:42.689165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:42.689179 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:42.691687 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:42.692179 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:42.692183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:42.692183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:42.692303 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:42.692496 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:42.692497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:42.692497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:42.692780 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:42.692930 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:42.693185 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 12:21:42.693357 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":68,"threads_started":1}
I20260928 12:21:42.715097 8427 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:21:42.715706 8427 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:21:42.720254 8427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58: Generated new TSK 0
I20260928 12:21:42.720454 8427 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.1.keytab.
Entry for principal oryx/127.2.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.1.keytab.
I20260928 12:21:42.748201 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.1:0
--local_ip_for_outbound_sockets=127.2.56.1
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:43477
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.1.keytab
--principal=oryx/127.2.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:43591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:42.837993 8442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:42.838220 8442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:42.838251 8442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:42.839404 8442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:42.839447 8442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:42.839481 8442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.1
I20260928 12:21:42.840963 8442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:43591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.1.keytab
--principal=oryx/127.2.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:43477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.1
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:42.841207 8442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:42.841457 8442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:42.846662 8447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:42.846745 8448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:42.846750 8450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:42.846920 8442 server_base.cc:1034] running on GCE node
I20260928 12:21:42.847119 8442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:42.847373 8442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:42.848524 8442 hybrid_clock.cc:648] HybridClock initialized: now 1790598102848506 us; error 34 us; skew 500 ppm
Sep 28 12:21:42 dist-test-slave-920p krb5kdc[8335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598102, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:42.851255 8442 init.cc:377] Logged in from keytab as oryx/127.2.56.1@KRBTEST.COM (short username oryx)
I20260928 12:21:42.851733 8442 webserver.cc:460] Webserver started at http://127.2.56.1:45751/ using document root <none> and password file <none>
I20260928 12:21:42.851985 8442 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:42.852036 8442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:42.852169 8442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:42.853432 8442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-0/data/instance:
uuid: "53c647c83fb847d090c0cc0c1e3e6f60"
format_stamp: "Formatted at 2026-09-28 12:21:42 on dist-test-slave-920p"
server_key: "894bf78498255acfad95a7a45c8e207c"
server_key_iv: "3415d5e41690eaeb9b3d5b5d827d272d"
server_key_version: "encryptionkey@0"
I20260928 12:21:42.853793 8442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance:
uuid: "53c647c83fb847d090c0cc0c1e3e6f60"
format_stamp: "Formatted at 2026-09-28 12:21:42 on dist-test-slave-920p"
server_key: "894bf78498255acfad95a7a45c8e207c"
server_key_iv: "3415d5e41690eaeb9b3d5b5d827d272d"
server_key_version: "encryptionkey@0"
I20260928 12:21:42.855141 8442 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 12:21:42.855825 8457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:42.855980 8442 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:21:42.856057 8442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-0/wal
uuid: "53c647c83fb847d090c0cc0c1e3e6f60"
format_stamp: "Formatted at 2026-09-28 12:21:42 on dist-test-slave-920p"
server_key: "894bf78498255acfad95a7a45c8e207c"
server_key_iv: "3415d5e41690eaeb9b3d5b5d827d272d"
server_key_version: "encryptionkey@0"
I20260928 12:21:42.856117 8442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:42.869199 8442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:42.871145 8442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:42.871291 8442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:42.871548 8442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:21:42.871886 8442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:42.871919 8442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:42.871951 8442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:42.871973 8442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:42.878561 8442 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.1:42075
I20260928 12:21:42.878621 8570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.1:42075 every 8 connection(s)
I20260928 12:21:42.878983 8442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
I20260928 12:21:42.883962 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 8442
I20260928 12:21:42.884068 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance
I20260928 12:21:42.884270 2272 external_mini_cluster.cc:1428] Setting key a361ddaeb20f70e587bf8d8e76a40a56
Sep 28 12:21:42 dist-test-slave-920p krb5kdc[8335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598102, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM
I20260928 12:21:42.889103 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:42.879755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:59637 (local address 127.2.56.62:43477)
0928 12:21:42.879804 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:42.879807 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:42.880010 (+ 203us) server_negotiation.cc:407] Connection header received
0928 12:21:42.880567 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:42.880568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:42.880580 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:42.880630 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:42.881696 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:42.882249 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:42.883256 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:42.883460 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:42.883505 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:42.886186 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:42.886188 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:42.886188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:42.886196 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:42.887684 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:42.888101 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:42.888102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:42.888103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:42.888171 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:42.888437 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:42.888437 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:42.888438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:42.888576 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:42.888640 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:42.888894 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 12:21:42.888994 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 12:21:42.889333 8573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:42.879724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.2.56.1:59637)
0928 12:21:42.879890 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:42.879932 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:42.880479 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:42.880696 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:42.880700 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:42.881364 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:42.881609 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:42.881616 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:42.882317 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:42.882317 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:42.883191 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:42.883193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:42.883516 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:42.883517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:42.883623 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:42.884244 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:42.884254 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:42.886097 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:42.887782 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:42.887789 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:42.887791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:42.888026 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:42.888253 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:42.888254 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:42.888255 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:42.888376 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:42.888632 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:42.888637 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:42.888774 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 12:21:42.888861 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 12:21:42.889008 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":90,"threads_started":1}
I20260928 12:21:42.889999 8571 heartbeater.cc:347] Connected to a master server at 127.2.56.62:43477
I20260928 12:21:42.890095 8571 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:42.890287 8571 heartbeater.cc:511] Master 127.2.56.62:43477 requested a full tablet report, sending...
I20260928 12:21:42.890856 8380 ts_manager.cc:194] Registered new tserver with Master: 53c647c83fb847d090c0cc0c1e3e6f60 (127.2.56.1:42075)
I20260928 12:21:42.891561 8380 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.56.1@KRBTEST.COM'} at 127.2.56.1:59637
WARNING: no policy specified for oryx/127.2.56.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.56.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.2.keytab.
Entry for principal oryx/127.2.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.2.keytab.
I20260928 12:21:42.910434 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.2:0
--local_ip_for_outbound_sockets=127.2.56.2
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:43477
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.2.keytab
--principal=oryx/127.2.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:43591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:42.997430 8576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:42.997691 8576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:42.997730 8576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:42.999099 8576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:42.999148 8576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:42.999176 8576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.2
I20260928 12:21:43.000646 8576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:43591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.2.keytab
--principal=oryx/127.2.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:43477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.2
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:43.000862 8576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:43.001072 8576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:43.006331 8581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:43.006347 8584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:43.006357 8582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:43.006596 8576 server_base.cc:1034] running on GCE node
I20260928 12:21:43.006817 8576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:43.007066 8576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:43.008208 8576 hybrid_clock.cc:648] HybridClock initialized: now 1790598103008195 us; error 30 us; skew 500 ppm
Sep 28 12:21:43 dist-test-slave-920p krb5kdc[8335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598103, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:43.010885 8576 init.cc:377] Logged in from keytab as oryx/127.2.56.2@KRBTEST.COM (short username oryx)
I20260928 12:21:43.011369 8576 webserver.cc:460] Webserver started at http://127.2.56.2:40811/ using document root <none> and password file <none>
I20260928 12:21:43.011608 8576 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:43.011658 8576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:43.011792 8576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:43.012991 8576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-1/data/instance:
uuid: "9c6a797c45334b15bca72df411c5d1bb"
format_stamp: "Formatted at 2026-09-28 12:21:43 on dist-test-slave-920p"
server_key: "5367e319c4e4b609edcd78e801d4ad16"
server_key_iv: "e07e542192ebaf1b339d88a6fd4291e1"
server_key_version: "encryptionkey@0"
I20260928 12:21:43.013394 8576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance:
uuid: "9c6a797c45334b15bca72df411c5d1bb"
format_stamp: "Formatted at 2026-09-28 12:21:43 on dist-test-slave-920p"
server_key: "5367e319c4e4b609edcd78e801d4ad16"
server_key_iv: "e07e542192ebaf1b339d88a6fd4291e1"
server_key_version: "encryptionkey@0"
I20260928 12:21:43.014775 8576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:21:43.015509 8591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:43.015655 8576 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:21:43.015738 8576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-1/wal
uuid: "9c6a797c45334b15bca72df411c5d1bb"
format_stamp: "Formatted at 2026-09-28 12:21:43 on dist-test-slave-920p"
server_key: "5367e319c4e4b609edcd78e801d4ad16"
server_key_iv: "e07e542192ebaf1b339d88a6fd4291e1"
server_key_version: "encryptionkey@0"
I20260928 12:21:43.015799 8576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:43.046553 8576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:43.048553 8576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:43.048704 8576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:43.048981 8576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:21:43.049371 8576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:43.049409 8576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:43.049441 8576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:43.049465 8576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:43.055656 8576 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.2:34785
I20260928 12:21:43.055718 8704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.2:34785 every 8 connection(s)
I20260928 12:21:43.056049 8576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
Sep 28 12:21:43 dist-test-slave-920p krb5kdc[8335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598103, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM
I20260928 12:21:43.065191 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.056763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:52161 (local address 127.2.56.62:43477)
0928 12:21:43.056844 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:43.056847 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:43.057051 (+ 204us) server_negotiation.cc:407] Connection header received
0928 12:21:43.057673 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:43.057675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:43.057688 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:43.057743 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:43.058731 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.059300 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.060409 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.060605 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.060653 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.062783 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:43.062784 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:43.062785 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:43.062791 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:43.064064 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.064364 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.064365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.064365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.064410 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.064586 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.064586 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.064587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.064682 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:43.064727 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:43.065009 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 12:21:43.065093 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 12:21:43.065376 8707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.056743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.2.56.2:52161)
0928 12:21:43.056922 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:43.056962 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:43.057546 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:43.057809 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:43.057813 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:43.058439 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:43.058663 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.058668 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.059373 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.059374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.060334 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.060336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.060660 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.060660 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.060762 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.061241 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:43.061251 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:43.062697 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:43.064121 (+ 1424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:43.064124 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:43.064126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:43.064315 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:43.064461 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:43.064462 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:43.064462 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:43.064546 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:43.064720 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:43.064723 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:43.064841 (+ 118us) client_negotiation.cc:769] Sending connection context
0928 12:21:43.064962 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 12:21:43.065120 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
I20260928 12:21:43.065497 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 8576
I20260928 12:21:43.065590 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance
I20260928 12:21:43.065760 2272 external_mini_cluster.cc:1428] Setting key 794dc933eece9c23c7e752c22bfe873c
I20260928 12:21:43.066009 8705 heartbeater.cc:347] Connected to a master server at 127.2.56.62:43477
I20260928 12:21:43.066116 8705 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:43.066320 8705 heartbeater.cc:511] Master 127.2.56.62:43477 requested a full tablet report, sending...
I20260928 12:21:43.066744 8380 ts_manager.cc:194] Registered new tserver with Master: 9c6a797c45334b15bca72df411c5d1bb (127.2.56.2:34785)
I20260928 12:21:43.067138 8380 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.56.2@KRBTEST.COM'} at 127.2.56.2:52161
WARNING: no policy specified for oryx/127.2.56.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.56.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.3.keytab.
Entry for principal oryx/127.2.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.3.keytab.
I20260928 12:21:43.089041 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.3:0
--local_ip_for_outbound_sockets=127.2.56.3
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:43477
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.3.keytab
--principal=oryx/127.2.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:43591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:43.178669 8710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:43.178916 8710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:43.178949 8710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:43.180123 8710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:43.180162 8710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:43.180197 8710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.3
I20260928 12:21:43.181689 8710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:43591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.3.keytab
--principal=oryx/127.2.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:43477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.3
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:43.181936 8710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:43.182156 8710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:43.187193 8716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:43.187193 8718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:43.187233 8715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:43.187367 8710 server_base.cc:1034] running on GCE node
I20260928 12:21:43.187585 8710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:43.187777 8710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:43.188906 8710 hybrid_clock.cc:648] HybridClock initialized: now 1790598103188895 us; error 28 us; skew 500 ppm
Sep 28 12:21:43 dist-test-slave-920p krb5kdc[8335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598103, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:43.191675 8710 init.cc:377] Logged in from keytab as oryx/127.2.56.3@KRBTEST.COM (short username oryx)
I20260928 12:21:43.192113 8710 webserver.cc:460] Webserver started at http://127.2.56.3:38823/ using document root <none> and password file <none>
I20260928 12:21:43.192308 8710 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:43.192346 8710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:43.192456 8710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:43.193583 8710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-2/data/instance:
uuid: "cf83a5232738461e9eace3fc21dd0929"
format_stamp: "Formatted at 2026-09-28 12:21:43 on dist-test-slave-920p"
server_key: "d19d7a7aef0e6b9a4cf24b50f7cb56d4"
server_key_iv: "e089f4805cf47704530b7f7c65c4504c"
server_key_version: "encryptionkey@0"
I20260928 12:21:43.193921 8710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance:
uuid: "cf83a5232738461e9eace3fc21dd0929"
format_stamp: "Formatted at 2026-09-28 12:21:43 on dist-test-slave-920p"
server_key: "d19d7a7aef0e6b9a4cf24b50f7cb56d4"
server_key_iv: "e089f4805cf47704530b7f7c65c4504c"
server_key_version: "encryptionkey@0"
I20260928 12:21:43.195168 8710 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:21:43.195812 8725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:43.195968 8710 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:43.196048 8710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-2/wal
uuid: "cf83a5232738461e9eace3fc21dd0929"
format_stamp: "Formatted at 2026-09-28 12:21:43 on dist-test-slave-920p"
server_key: "d19d7a7aef0e6b9a4cf24b50f7cb56d4"
server_key_iv: "e089f4805cf47704530b7f7c65c4504c"
server_key_version: "encryptionkey@0"
I20260928 12:21:43.196102 8710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:43.215524 8710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:43.217602 8710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:43.217726 8710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:43.217952 8710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:21:43.218294 8710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:43.218339 8710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:43.218364 8710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:43.218374 8710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:43.224879 8710 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.3:43087
I20260928 12:21:43.224948 8838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.3:43087 every 8 connection(s)
I20260928 12:21:43.225301 8710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
Sep 28 12:21:43 dist-test-slave-920p krb5kdc[8335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598103, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM
I20260928 12:21:43.234298 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 8710
I20260928 12:21:43.234398 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance
I20260928 12:21:43.234580 2272 external_mini_cluster.cc:1428] Setting key fbb75050c52441b066d8617adde17cfe
I20260928 12:21:43.234808 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.226131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:60123 (local address 127.2.56.62:43477)
0928 12:21:43.226213 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:43.226216 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:43.226410 (+ 194us) server_negotiation.cc:407] Connection header received
0928 12:21:43.227028 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:43.227029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:43.227041 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:43.227081 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:43.228131 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.228667 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.229744 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.229956 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.230010 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.232222 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:43.232224 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:43.232224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:43.232231 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:43.233577 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.233899 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.233899 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.233900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.233948 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.234130 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.234131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.234131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.234274 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:43.234343 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:43.234623 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 12:21:43.234704 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 12:21:43.235038 8841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.226108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.2.56.3:60123)
0928 12:21:43.226273 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:43.226318 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:43.226922 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:43.227155 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:43.227159 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:43.227818 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:43.228053 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.228058 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.228733 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.228734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.229671 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.229674 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.230025 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.230026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.230135 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.230675 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:43.230685 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:43.232135 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:43.233633 (+ 1498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:43.233637 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:43.233639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:43.233847 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:43.233990 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:43.233991 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:43.233991 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:43.234084 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:43.234329 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:43.234332 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:43.234493 (+ 161us) client_negotiation.cc:769] Sending connection context
0928 12:21:43.234582 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 12:21:43.234754 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":86,"threads_started":1}
I20260928 12:21:43.235535 8839 heartbeater.cc:347] Connected to a master server at 127.2.56.62:43477
I20260928 12:21:43.235641 8839 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:43.235841 8839 heartbeater.cc:511] Master 127.2.56.62:43477 requested a full tablet report, sending...
I20260928 12:21:43.236222 8380 ts_manager.cc:194] Registered new tserver with Master: cf83a5232738461e9eace3fc21dd0929 (127.2.56.3:43087)
I20260928 12:21:43.236605 8380 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.56.3@KRBTEST.COM'} at 127.2.56.3:60123
I20260928 12:21:43.246289 2272 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.56.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.56.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.56.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.61.keytab.
Entry for principal oryx/127.2.56.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.61.keytab.
I20260928 12:21:43.269685 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.61:42657
--webserver_interface=127.2.56.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.61.keytab
--principal=oryx/127.2.56.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:43591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:43477,127.2.56.61:42657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:43.354705 8844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:43.354931 8844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:43.354954 8844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:43.356081 8844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:21:43.356118 8844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:43.356132 8844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:43.356142 8844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:21:43.356153 8844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:21:43.357626 8844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:43591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:43477,127.2.56.61:42657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.61.keytab
--principal=oryx/127.2.56.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.61:42657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.56.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
W20260928 12:21:43.357733 8844 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.2.56.62:43477,127.2.56.61:42657'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 12:21:43.357847 8844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:43.358032 8844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:43.362815 8849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:43.362809 8852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:43.362861 8850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:43.363106 8844 server_base.cc:1034] running on GCE node
I20260928 12:21:43.363315 8844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:43.363577 8844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:43.364708 8844 hybrid_clock.cc:648] HybridClock initialized: now 1790598103364697 us; error 30 us; skew 500 ppm
Sep 28 12:21:43 dist-test-slave-920p krb5kdc[8335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598103, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:43.367179 8844 init.cc:377] Logged in from keytab as oryx/127.2.56.61@KRBTEST.COM (short username oryx)
I20260928 12:21:43.367617 8844 webserver.cc:460] Webserver started at http://127.2.56.61:34111/ using document root <none> and password file <none>
I20260928 12:21:43.367816 8844 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:43.367854 8844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:43.367959 8844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:43.368989 8844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-1/data/instance:
uuid: "2b6ccb471f0e419c91dd33e886b9fe85"
format_stamp: "Formatted at 2026-09-28 12:21:43 on dist-test-slave-920p"
server_key: "03efcfbe65e86e1e4993132bb6cb6425"
server_key_iv: "a8d890c9d982fcd5b43981556ed33411"
server_key_version: "encryptionkey@0"
I20260928 12:21:43.369352 8844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-1/wal/instance:
uuid: "2b6ccb471f0e419c91dd33e886b9fe85"
format_stamp: "Formatted at 2026-09-28 12:21:43 on dist-test-slave-920p"
server_key: "03efcfbe65e86e1e4993132bb6cb6425"
server_key_iv: "a8d890c9d982fcd5b43981556ed33411"
server_key_version: "encryptionkey@0"
I20260928 12:21:43.370623 8844 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:21:43.371300 8859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:43.371454 8844 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:43.371536 8844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-1/wal
uuid: "2b6ccb471f0e419c91dd33e886b9fe85"
format_stamp: "Formatted at 2026-09-28 12:21:43 on dist-test-slave-920p"
server_key: "03efcfbe65e86e1e4993132bb6cb6425"
server_key_iv: "a8d890c9d982fcd5b43981556ed33411"
server_key_version: "encryptionkey@0"
I20260928 12:21:43.371598 8844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:43.386046 8844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:43.387899 8844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:43.388046 8844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:43.392371 8844 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.61:42657
I20260928 12:21:43.392416 8911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.61:42657 every 8 connection(s)
I20260928 12:21:43.392768 8844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-1/data/info.pb
I20260928 12:21:43.393800 8912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:43.394170 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 8844
I20260928 12:21:43.396405 8912 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 12:21:43 dist-test-slave-920p krb5kdc[8335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.56.61@KRBTEST.COM
Sep 28 12:21:43 dist-test-slave-920p krb5kdc[8335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598103, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.61@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM
I20260928 12:21:43.404650 8915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.394840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35256 (local address 127.2.56.61:42657)
0928 12:21:43.395000 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:43.395009 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:43.395036 (+ 27us) server_negotiation.cc:407] Connection header received
0928 12:21:43.395514 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:43.395522 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:43.395578 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:43.395704 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:43.396460 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.397640 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.398623 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.399233 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.399339 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.401715 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:43.401723 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:43.401724 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:43.401731 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:43.403205 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.403561 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.403563 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.403564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.403634 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.403809 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.403809 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.403810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.404082 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:43.404220 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:43.404322 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 12:21:43.404423 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20260928 12:21:43.404850 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.397091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51696 (local address 127.2.56.62:43477)
0928 12:21:43.397144 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:43.397146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:43.397286 (+ 140us) server_negotiation.cc:407] Connection header received
0928 12:21:43.397362 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:43.397363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:43.397373 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:43.397424 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:43.397961 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.398695 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.399708 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.399947 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.399997 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.402247 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:43.402249 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:43.402249 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:43.402256 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:43.403652 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.404004 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.404004 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.404005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.404051 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.404212 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.404212 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.404213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.404411 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:43.404483 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:43.404687 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 12:21:43.404770 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 12:21:43.405028 8916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.397008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.0.0.1:51696)
0928 12:21:43.397164 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:43.397204 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:43.397306 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:43.397452 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:43.397455 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:43.397811 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:43.397887 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.397889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.398790 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.398791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.399632 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.399634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.400024 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.400025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.400112 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.400529 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:43.400537 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:43.402169 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:43.403716 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:43.403721 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:43.403723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:43.403937 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:43.404098 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:43.404099 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:43.404100 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:43.404160 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:43.404510 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:43.404516 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:43.404572 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 12:21:43.404634 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 12:21:43.404920 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
I20260928 12:21:43.405594 8912 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 12:21:43 dist-test-slave-920p krb5kdc[8335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598103, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.61@KRBTEST.COM for oryx/127.2.56.61@KRBTEST.COM
I20260928 12:21:43.410902 8916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.405861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.61:42657 (local address 127.0.0.1:35260)
0928 12:21:43.405900 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:43.405917 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:43.405954 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:43.406067 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:43.406067 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:43.406184 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:43.406228 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.406230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.406811 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.406812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.407281 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.407282 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.407600 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.407601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.407661 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.407951 (+ 290us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:43.407954 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:43.408957 (+ 1003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:43.410098 (+ 1141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:43.410099 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:43.410100 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:43.410239 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:43.410379 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:43.410380 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:43.410380 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:43.410431 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:43.410614 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:43.410615 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:43.410648 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 12:21:43.410707 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 12:21:43.410795 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 12:21:43.410951 8915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.405922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35260 (local address 127.2.56.61:42657)
0928 12:21:43.405954 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:43.405955 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:43.405962 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:21:43.406014 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:43.406015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:43.406023 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:43.406065 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:43.406285 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.406752 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.407341 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.407548 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.407597 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.409023 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:43.409024 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:43.409025 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:43.409031 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:43.410036 (+ 1005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.410291 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.410292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.410292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.410332 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.410478 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.410478 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.410479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.410566 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:43.410615 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:43.410730 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 12:21:43.410842 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":18}
I20260928 12:21:43.412254 8912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85: Bootstrap starting.
I20260928 12:21:43.413100 8912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:43.413394 8912 log.cc:824] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:43.414081 8912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85: No bootstrap required, opened a new log
I20260928 12:21:43.415531 8912 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } }
I20260928 12:21:43.415658 8912 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:43.415685 8912 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b6ccb471f0e419c91dd33e886b9fe85, State: Initialized, Role: FOLLOWER
I20260928 12:21:43.415789 8912 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } }
I20260928 12:21:43.416029 8918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } } }
I20260928 12:21:43.416113 8912 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:21:43.416126 8918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 12:21:43.417682 8929 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:21:43.417728 8929 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:21:43.419236 8844 master_runner.cc:428] Detected that this master 2b6ccb471f0e419c91dd33e886b9fe85 is joining an existing cluster
I20260928 12:21:43.419277 8844 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.56.61:42657
I20260928 12:21:43.426640 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.420258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51706 (local address 127.2.56.62:43477)
0928 12:21:43.420555 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:43.420557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:43.420564 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:21:43.420586 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:43.420587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:43.420596 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:43.420645 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:43.421010 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.421696 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.422752 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.423015 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.423067 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.423905 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:43.423906 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:43.423907 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:43.423914 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:43.425532 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.425951 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.425953 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.425953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.426011 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.426172 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.426173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.426173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.426293 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:43.426350 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:43.426481 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 12:21:43.426546 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275}
I20260928 12:21:43.426677 8936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.420180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.0.0.1:51706)
0928 12:21:43.420362 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:43.420372 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:43.420432 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:43.420658 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:43.420659 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:43.420854 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:43.420948 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.420950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.421772 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.421773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.422680 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.422682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.423071 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.423072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.423133 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.423484 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:43.423488 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:43.423832 (+ 344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:43.425612 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:43.425613 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:43.425613 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:43.425878 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:43.426058 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:43.426058 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:43.426059 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:43.426108 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:43.426340 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:43.426341 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:43.426381 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 12:21:43.426440 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 12:21:43.426545 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":98,"threads_started":1}
I20260928 12:21:43.426807 8915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.420506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35262 (local address 127.2.56.61:42657)
0928 12:21:43.420551 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:43.420553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:43.420563 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:21:43.420585 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:43.420585 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:43.420594 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:43.420647 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:43.421369 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.421948 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.422612 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.422837 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.422893 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.423943 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:43.423943 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:43.423944 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:43.423949 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:43.425657 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.425989 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.425990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.425991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.426043 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.426212 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.426213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.426213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.426319 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:43.426358 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:43.426547 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 12:21:43.426618 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 12:21:43.427043 8935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.420193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.61:42657 (local address 127.0.0.1:35262)
0928 12:21:43.420507 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:43.420515 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:43.420552 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:43.421032 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:43.421032 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:43.421223 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:43.421307 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.421309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.422008 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.422009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.422549 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.422550 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.422910 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.422911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.422975 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.423376 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:43.423381 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:43.423849 (+ 468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:43.425716 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:43.425717 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:43.425717 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:43.425905 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:43.426106 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:43.426107 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:43.426107 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:43.426155 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:43.426398 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:43.426398 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:43.426441 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 12:21:43.426687 (+ 246us) client_negotiation.cc:240] Negotiation successful
0928 12:21:43.426760 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":70,"threads_started":1}
Sep 28 12:21:43 dist-test-slave-920p krb5kdc[8335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598102, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.62@KRBTEST.COM for oryx/127.2.56.61@KRBTEST.COM
I20260928 12:21:43.435606 8937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.428756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.61:42657 (local address 127.0.0.1:35266)
0928 12:21:43.428874 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:43.428931 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:43.428995 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:43.429123 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:43.429125 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:43.429407 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:43.429462 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.429463 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.430285 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.430285 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.430870 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.430871 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.431273 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.431273 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.431324 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.431709 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:43.431712 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:43.433220 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:43.434695 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:43.434701 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:43.434703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:43.434883 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:43.435054 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:43.435054 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:43.435054 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:43.435098 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:43.435310 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:43.435314 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:43.435355 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 12:21:43.435415 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 12:21:43.435516 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":73,"threads_started":1}
I20260928 12:21:43.435612 8915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.428799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35266 (local address 127.2.56.61:42657)
0928 12:21:43.428837 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:43.428839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:43.428988 (+ 149us) server_negotiation.cc:407] Connection header received
0928 12:21:43.429046 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:43.429047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:43.429056 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:43.429119 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:43.429518 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.430219 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.430929 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.431213 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.431276 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.433307 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:43.433308 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:43.433309 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:43.433315 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:43.434626 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.434945 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.434945 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.434946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.434999 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.435142 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.435142 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.435143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.435258 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:43.435321 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:43.435454 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 12:21:43.435523 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 12:21:43.436132 8380 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.56.61:42657: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85"
member_type: NON_VOTER
last_known_addr {
host: "127.2.56.61"
port: 42657
}
attrs {
promote: true
}
}
dest_uuid: "262a43ba51f54d5ba1fc806531019e58"
cas_config_opid_index: -1
I20260928 12:21:43.436463 8380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: NON_VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: true } }
I20260928 12:21:43.436939 8938 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 2b6ccb471f0e419c91dd33e886b9fe85 (127.2.56.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: NON_VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: true } } }
I20260928 12:21:43.437254 8886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:43.437958 8938 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.56.61:42657
I20260928 12:21:43.438163 8844 master.cc:537] Master@127.2.56.61:42657 shutting down...
I20260928 12:21:43.438285 8886 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [term 1 FOLLOWER]: Refusing update from remote peer 262a43ba51f54d5ba1fc806531019e58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 12:21:43.438612 8938 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: NON_VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:21:43.438937 8939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "262a43ba51f54d5ba1fc806531019e58" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: NON_VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: true } } }
I20260928 12:21:43.439019 8939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [sys.catalog]: This master's current role is: LEADER
W20260928 12:21:43.439977 8364 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 12:21:43.440325 8364 consensus_peers.cc:487] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 -> Peer 2b6ccb471f0e419c91dd33e886b9fe85 (127.2.56.61:42657): Couldn't send request to peer 2b6ccb471f0e419c91dd33e886b9fe85. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 12:21:43.440979 8844 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:21:43.441067 8844 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:43.441121 8844 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85: stopping tablet replica
I20260928 12:21:43.453838 8844 master.cc:552] Master@127.2.56.61:42657 shutdown complete.
I20260928 12:21:43.453896 8844 master_runner.cc:305] Clearing existing system tablet
I20260928 12:21:43.454035 8844 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:43.454083 8844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:43.454813 8844 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:43.455300 8943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:43.455451 8844 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:43.455506 8844 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:43.455528 8844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-1/wal
uuid: "2b6ccb471f0e419c91dd33e886b9fe85"
format_stamp: "Formatted at 2026-09-28 12:21:43 on dist-test-slave-920p"
server_key: "03efcfbe65e86e1e4993132bb6cb6425"
server_key_iv: "a8d890c9d982fcd5b43981556ed33411"
server_key_version: "encryptionkey@0"
I20260928 12:21:43.456758 8844 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:43.458204 8844 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 12:21:43.458261 8844 log.cc:1196] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 12:21:43.458447 8844 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85: Deleting consensus metadata
I20260928 12:21:43.458554 8844 master_runner.cc:315] Copying system tablet from 127.2.56.62:43477
I20260928 12:21:43.459157 8844 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85: tablet copy: Beginning tablet copy session from remote peer at address 127.2.56.62:43477
I20260928 12:21:43.465610 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.459501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51718 (local address 127.2.56.62:43477)
0928 12:21:43.459544 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:43.459546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:43.459665 (+ 119us) server_negotiation.cc:407] Connection header received
0928 12:21:43.459727 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:43.459728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:43.459740 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:43.459794 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:43.460204 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.460873 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.461989 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.462179 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.462225 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.463338 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:43.463340 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:43.463340 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:43.463347 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:43.464588 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.464949 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.464949 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.464949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.464991 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.465147 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.465148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.465148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.465245 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:43.465321 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:43.465478 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 12:21:43.465540 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:21:43.465778 8949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.459451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.0.0.1:51718)
0928 12:21:43.459593 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:43.459602 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:43.459670 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:43.459828 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:43.459829 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:43.460047 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:43.460123 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.460126 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.460945 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.460946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.461910 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.461913 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.462236 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.462237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.462309 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.462818 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:43.462823 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:43.463245 (+ 422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:43.464656 (+ 1411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:43.464657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:43.464658 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:43.464877 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:43.465038 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:43.465038 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:43.465039 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:43.465100 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:43.465314 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:43.465315 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:43.465362 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 12:21:43.465452 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 12:21:43.465557 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":75,"threads_started":1}
I20260928 12:21:43.466136 8400 tablet_copy_service.cc:140] P 262a43ba51f54d5ba1fc806531019e58: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2b6ccb471f0e419c91dd33e886b9fe85 ({username='oryx', principal='oryx/127.2.56.61@KRBTEST.COM'} at 127.0.0.1:51718)
I20260928 12:21:43.466238 8400 tablet_copy_service.cc:161] P 262a43ba51f54d5ba1fc806531019e58: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2b6ccb471f0e419c91dd33e886b9fe85 at {username='oryx', principal='oryx/127.2.56.61@KRBTEST.COM'} at 127.0.0.1:51718: session id = 2b6ccb471f0e419c91dd33e886b9fe85-00000000000000000000000000000000
I20260928 12:21:43.467007 8400 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 12:21:43.467608 8844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 12:21:43.468555 8844 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85: tablet copy: Starting download of 0 data blocks...
I20260928 12:21:43.468690 8844 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85: tablet copy: Starting download of 1 WAL segments...
I20260928 12:21:43.469517 8844 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 12:21:43.470259 8400 tablet_copy_service.cc:342] P 262a43ba51f54d5ba1fc806531019e58: Request end of tablet copy session 2b6ccb471f0e419c91dd33e886b9fe85-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.56.61@KRBTEST.COM'} at 127.0.0.1:51718
I20260928 12:21:43.470306 8400 tablet_copy_service.cc:433] P 262a43ba51f54d5ba1fc806531019e58: ending tablet copy session 2b6ccb471f0e419c91dd33e886b9fe85-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2b6ccb471f0e419c91dd33e886b9fe85
I20260928 12:21:43.472726 8844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:43.474026 8950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:43.474073 8951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:43.474243 8844 server_base.cc:1034] running on GCE node
W20260928 12:21:43.477380 8953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:43.477566 8844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:43.477716 8844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:43.478826 8844 hybrid_clock.cc:648] HybridClock initialized: now 1790598103478814 us; error 32 us; skew 500 ppm
Sep 28 12:21:43 dist-test-slave-920p krb5kdc[8335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598103, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:43.480306 8844 init.cc:377] Logged in from keytab as oryx/127.2.56.61@KRBTEST.COM (short username oryx)
I20260928 12:21:43.480607 8844 webserver.cc:460] Webserver started at http://127.2.56.61:32861/ using document root <none> and password file <none>
I20260928 12:21:43.480715 8844 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:43.480746 8844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:43.481449 8844 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:21:43.481966 8960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:43.482070 8844 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:21:43.482120 8844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-1/wal
uuid: "2b6ccb471f0e419c91dd33e886b9fe85"
format_stamp: "Formatted at 2026-09-28 12:21:43 on dist-test-slave-920p"
server_key: "03efcfbe65e86e1e4993132bb6cb6425"
server_key_iv: "a8d890c9d982fcd5b43981556ed33411"
server_key_version: "encryptionkey@0"
I20260928 12:21:43.482187 8844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:43.487500 8844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:43.488910 8844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:43.492515 8844 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.61:42657
I20260928 12:21:43.492849 9012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.61:42657 every 8 connection(s)
I20260928 12:21:43.492941 8844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-1/data/info.pb
I20260928 12:21:43.493773 9013 sys_catalog.cc:263] Verifying existing consensus state
I20260928 12:21:43.494168 9013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85: Bootstrap starting.
I20260928 12:21:43.496488 9013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:21:43.496671 9013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85: Bootstrap complete.
I20260928 12:21:43.496872 9013 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: NON_VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: true } }
I20260928 12:21:43.496948 9013 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2b6ccb471f0e419c91dd33e886b9fe85, State: Initialized, Role: LEARNER
I20260928 12:21:43.496984 9013 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: NON_VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: true } }
I20260928 12:21:43.497151 9013 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:21:43.497140 9016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: NON_VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: true } } }
I20260928 12:21:43.497218 9016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [sys.catalog]: This master's current role is: LEARNER
I20260928 12:21:43.498564 9027 catalog_manager.cc:1259] Loaded cluster ID: f5f2d04c9c0b48a1b4ff09425dfdc0bd
I20260928 12:21:43.498607 9027 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85: loading cluster ID for follower catalog manager: success
I20260928 12:21:43.499612 9027 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85: acquiring CA information for follower catalog manager: success
I20260928 12:21:43.499886 9027 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 12:21:43.512388 9028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.506365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35280 (local address 127.2.56.61:42657)
0928 12:21:43.506508 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:43.506510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:43.506518 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:21:43.506543 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:43.506544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:43.506553 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:43.506612 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:43.507156 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.507820 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.508548 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.508751 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.508807 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.509996 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:43.509997 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:43.509998 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:43.510004 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:43.511380 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.511682 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.511682 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.511683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.511736 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.511892 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.511892 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.511893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.512005 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:43.512101 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:43.512193 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 12:21:43.512271 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for oryx/127.2.56.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.56.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.56.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.60.keytab.
Entry for principal oryx/127.2.56.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.60.keytab.
I20260928 12:21:43.540376 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.60:44031
--webserver_interface=127.2.56.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.60.keytab
--principal=oryx/127.2.56.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:43591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:43477,127.2.56.61:42657,127.2.56.60:44031
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:43.643407 9031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:43.643633 9031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:43.643654 9031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:43.644814 9031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:21:43.644860 9031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:43.644872 9031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:43.644883 9031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:21:43.644893 9031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:21:43.646430 9031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:43591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:43477,127.2.56.61:42657,127.2.56.60:44031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.60.keytab
--principal=oryx/127.2.56.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.60:44031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.56.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:43.646673 9031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:43.646885 9031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:21:43.651898 9031 server_base.cc:1034] running on GCE node
W20260928 12:21:43.651860 9036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:43.651839 9039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:43.651835 9037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:43.652354 9031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:43.652673 9031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:43.653820 9031 hybrid_clock.cc:648] HybridClock initialized: now 1790598103653809 us; error 37 us; skew 500 ppm
Sep 28 12:21:43 dist-test-slave-920p krb5kdc[8335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598103, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:43.656364 9031 init.cc:377] Logged in from keytab as oryx/127.2.56.60@KRBTEST.COM (short username oryx)
I20260928 12:21:43.656788 9031 webserver.cc:460] Webserver started at http://127.2.56.60:38379/ using document root <none> and password file <none>
I20260928 12:21:43.657001 9031 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:43.657042 9031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:43.657140 9031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:43.658341 9031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-2/data/instance:
uuid: "35a79f2366a84381b917b76c334e55bc"
format_stamp: "Formatted at 2026-09-28 12:21:43 on dist-test-slave-920p"
server_key: "1994258b87284b10bef20b9c673923f4"
server_key_iv: "3d36bb5581b8c7002f71dbb3cefac854"
server_key_version: "encryptionkey@0"
I20260928 12:21:43.658694 9031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-2/wal/instance:
uuid: "35a79f2366a84381b917b76c334e55bc"
format_stamp: "Formatted at 2026-09-28 12:21:43 on dist-test-slave-920p"
server_key: "1994258b87284b10bef20b9c673923f4"
server_key_iv: "3d36bb5581b8c7002f71dbb3cefac854"
server_key_version: "encryptionkey@0"
I20260928 12:21:43.659997 9031 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:21:43.660646 9046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:43.660796 9031 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:43.660877 9031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-2/wal
uuid: "35a79f2366a84381b917b76c334e55bc"
format_stamp: "Formatted at 2026-09-28 12:21:43 on dist-test-slave-920p"
server_key: "1994258b87284b10bef20b9c673923f4"
server_key_iv: "3d36bb5581b8c7002f71dbb3cefac854"
server_key_version: "encryptionkey@0"
I20260928 12:21:43.660938 9031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:43.696652 9031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:43.699342 9031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:43.699487 9031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:43.703765 9031 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.60:44031
I20260928 12:21:43.703802 9098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.60:44031 every 8 connection(s)
I20260928 12:21:43.704169 9031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-2/data/info.pb
I20260928 12:21:43.705219 9099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:43.706032 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 9031
I20260928 12:21:43.707285 9099 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 12:21:43 dist-test-slave-920p krb5kdc[8335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598103, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.60@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM
Sep 28 12:21:43 dist-test-slave-920p krb5kdc[8335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.56.60@KRBTEST.COM
I20260928 12:21:43.715651 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.708311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51720 (local address 127.2.56.62:43477)
0928 12:21:43.708401 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:43.708404 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:43.708447 (+ 43us) server_negotiation.cc:407] Connection header received
0928 12:21:43.708519 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:43.708520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:43.708533 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:43.708572 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:43.709149 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.709678 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.710596 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.710776 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.710832 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.712998 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:43.713000 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:43.713000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:43.713007 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:43.714301 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.714729 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.714730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.714731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.714783 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.715003 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.715004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.715004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.715137 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:43.715204 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:43.715478 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 12:21:43.715557 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 12:21:43.715840 9102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.708169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.0.0.1:51720)
0928 12:21:43.708344 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:43.708384 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:43.708459 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:43.708602 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:43.708604 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:43.708955 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:43.709075 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.709080 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.709758 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.709759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.710521 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.710523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.710841 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.710841 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.710902 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.711440 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:43.711449 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:43.712915 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:43.714375 (+ 1460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:43.714381 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:43.714384 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:43.714661 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:43.714834 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:43.714834 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:43.714835 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:43.714949 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:43.715206 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:43.715211 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:43.715345 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 12:21:43.715452 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 12:21:43.715605 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":108,"threads_started":1}
I20260928 12:21:43.716173 9101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.706796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44120 (local address 127.2.56.60:44031)
0928 12:21:43.706986 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:43.706998 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:43.707034 (+ 36us) server_negotiation.cc:407] Connection header received
0928 12:21:43.707560 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:43.707570 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:43.707642 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:43.707806 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:43.708534 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.709697 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.710378 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.710764 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.710857 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.713548 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:43.713554 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:43.713555 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:43.713560 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:43.715003 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.715320 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.715323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.715324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.715378 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.715559 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.715560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.715560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.715705 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:43.715828 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:43.716023 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 12:21:43.716088 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":80,"threads_started":1}
I20260928 12:21:43.716459 9099 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 12:21:43 dist-test-slave-920p krb5kdc[8335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598103, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.60@KRBTEST.COM for oryx/127.2.56.61@KRBTEST.COM
I20260928 12:21:43.723052 9102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.717570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.61:42657 (local address 127.0.0.1:35288)
0928 12:21:43.717600 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:43.717609 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:43.717652 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:43.717836 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:43.717836 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:43.717947 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:43.717990 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.717992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.718585 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.718585 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.719120 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.719122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.719422 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.719422 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.719468 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.719744 (+ 276us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:43.719747 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:43.720827 (+ 1080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:43.722218 (+ 1391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:43.722219 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:43.722219 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:43.722364 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:43.722527 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:43.722528 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:43.722528 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:43.722581 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:43.722783 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:43.722784 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:43.722824 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 12:21:43.722882 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 12:21:43.722961 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20260928 12:21:43.723088 9028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.717672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35288 (local address 127.2.56.61:42657)
0928 12:21:43.717734 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:43.717736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:43.717744 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:21:43.717771 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:43.717771 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:43.717786 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:43.717827 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:43.718040 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.718534 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.719180 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.719375 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.719422 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.720904 (+ 1482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:43.720905 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:43.720906 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:43.720912 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:43.722141 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.722427 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.722428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.722429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.722478 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.722627 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.722628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.722628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.722732 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:43.722786 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:43.722910 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 12:21:43.722999 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 12:21:43.723497 9099 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.56.60" port: 44031 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 12:21:43 dist-test-slave-920p krb5kdc[8335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598103, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.60@KRBTEST.COM for oryx/127.2.56.60@KRBTEST.COM
I20260928 12:21:43.728428 9102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.723718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:44031 (local address 127.0.0.1:44136)
0928 12:21:43.723756 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:43.723764 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:43.723802 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:43.723894 (+ 92us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:43.723894 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:43.723990 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:43.724016 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.724017 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.724555 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.724555 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.724997 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.724998 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.725305 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.725306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.725356 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.725632 (+ 276us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:43.725635 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:43.726638 (+ 1003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:43.727731 (+ 1093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:43.727731 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:43.727732 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:43.727881 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:43.728015 (+ 134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:43.728015 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:43.728015 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:43.728056 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:43.728204 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:43.728204 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:43.728233 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 12:21:43.728274 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 12:21:43.728341 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 12:21:43.728469 9101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.723760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44136 (local address 127.2.56.60:44031)
0928 12:21:43.723790 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:43.723791 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:43.723799 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:21:43.723844 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:43.723845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:43.723852 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:43.723890 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:43.724059 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.724508 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.725043 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.725226 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.725300 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.726702 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:43.726703 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:43.726704 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:43.726709 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:43.727667 (+ 958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.727935 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.727936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.727936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.727975 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.728089 (+ 114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.728089 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.728089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.728169 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:43.728207 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:43.728289 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 12:21:43.728380 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":14}
I20260928 12:21:43.729923 9099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc: Bootstrap starting.
I20260928 12:21:43.730818 9099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:43.731092 9099 log.cc:824] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:43.731805 9099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc: No bootstrap required, opened a new log
I20260928 12:21:43.733362 9099 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 44031 } }
I20260928 12:21:43.733518 9099 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:43.733554 9099 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35a79f2366a84381b917b76c334e55bc, State: Initialized, Role: FOLLOWER
I20260928 12:21:43.733680 9099 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 44031 } }
I20260928 12:21:43.733944 9104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 44031 } } }
I20260928 12:21:43.734006 9099 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:21:43.734025 9104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [sys.catalog]: This master's current role is: FOLLOWER
W20260928 12:21:43.735575 9115 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:21:43.735618 9115 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:21:43.737896 9031 master_runner.cc:428] Detected that this master 35a79f2366a84381b917b76c334e55bc is joining an existing cluster
I20260928 12:21:43.737952 9031 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.56.60:44031
I20260928 12:21:43.745728 9123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.739043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.61:42657 (local address 127.0.0.1:35300)
0928 12:21:43.739599 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:43.739608 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:43.739663 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:43.739773 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:43.739774 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:43.739950 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:43.740010 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.740011 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.740987 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.740988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.741887 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.741889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.742393 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.742394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.742466 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.742916 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:43.742921 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:43.743246 (+ 325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:43.744833 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:43.744834 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:43.744834 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:43.744989 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:43.745170 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:43.745170 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:43.745170 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:43.745213 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:43.745482 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:43.745483 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:43.745521 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 12:21:43.745569 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 12:21:43.745645 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":86,"threads_started":1}
I20260928 12:21:43.745774 9028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.739233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35300 (local address 127.2.56.61:42657)
0928 12:21:43.739545 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:43.739547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:43.739671 (+ 124us) server_negotiation.cc:407] Connection header received
0928 12:21:43.739702 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:43.739703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:43.739714 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:43.739773 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:43.740092 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.740900 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.741975 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.742280 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.742319 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.743336 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:43.743338 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:43.743338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:43.743346 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:43.744766 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.745049 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.745050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.745050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.745114 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.745249 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.745249 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.745250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.745425 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:43.745496 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:43.745596 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 12:21:43.745679 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288}
I20260928 12:21:43.768456 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 8442
I20260928 12:21:43.769058 9101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.739499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44142 (local address 127.2.56.60:44031)
0928 12:21:43.739533 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:43.739534 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:43.740015 (+ 481us) server_negotiation.cc:407] Connection header received
0928 12:21:43.740086 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:43.740087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:43.740094 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:43.740119 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:43.740426 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.741108 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.742054 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.742316 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.742345 (+ 29us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.766507 (+ 24162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:43.766508 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:43.766509 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:43.766517 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:43.768021 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.768365 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.768366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.768366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.768409 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.768588 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.768589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.768589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.768725 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:43.768760 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:43.768893 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 12:21:43.768970 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 12:21:43.769397 9121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.739361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:44031 (local address 127.0.0.1:44142)
0928 12:21:43.739960 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:43.739968 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:43.740033 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:43.740160 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:43.740161 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:43.740310 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:43.740360 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.740361 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.741186 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.741187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.741984 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.741986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.742401 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.742402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.742455 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.765867 (+ 23412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:43.765876 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:43.766370 (+ 494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:43.768117 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:43.768118 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:43.768119 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:43.768298 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:43.768480 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:43.768480 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:43.768481 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:43.768530 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:43.768791 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:43.768791 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:43.768821 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 12:21:43.769172 (+ 351us) client_negotiation.cc:240] Negotiation successful
0928 12:21:43.769296 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":60,"threads_started":1}
I20260928 12:21:43.770422 9122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.739039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.0.0.1:51736)
0928 12:21:43.739197 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:43.739209 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:43.739286 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:43.739482 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:43.739483 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:43.739751 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:43.739843 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.739845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.740837 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.740838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.741824 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.741827 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.742377 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.742377 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.742451 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.765911 (+ 23460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:43.765924 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:43.766450 (+ 526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:43.768273 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:43.768273 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:43.768274 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:43.768447 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:43.769808 (+ 1361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:43.769809 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:43.769809 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:43.769861 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:43.770110 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:43.770111 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:43.770168 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 12:21:43.770217 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 12:21:43.770318 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":83,"threads_started":1}
I20260928 12:21:43.770447 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.739088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51736 (local address 127.2.56.62:43477)
0928 12:21:43.739140 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:43.739142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:43.739261 (+ 119us) server_negotiation.cc:407] Connection header received
0928 12:21:43.739361 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:43.739362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:43.739376 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:43.739412 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:43.739925 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.740740 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.741932 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.742238 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.742285 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.766564 (+ 24279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:43.766566 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:43.766566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:43.766575 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:43.768189 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.768534 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.768535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.768536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.768593 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.769928 (+ 1335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.769929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.769930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.770054 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:43.770113 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:43.770267 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 12:21:43.770347 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 12:21:43.776644 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.1:42075
--local_ip_for_outbound_sockets=127.2.56.1
--tserver_master_addrs=127.2.56.62:43477
--webserver_port=45751
--webserver_interface=127.2.56.1
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.1.keytab
--principal=oryx/127.2.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:43591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.56.62:43477,127.2.56.61:42657
--tserver_master_addrs=127.2.56.62:43477,127.2.56.61:42657,127.2.56.60:44031 with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 12:21:43 dist-test-slave-920p krb5kdc[8335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598102, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.62@KRBTEST.COM for oryx/127.2.56.60@KRBTEST.COM
I20260928 12:21:43.784102 8937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.777642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:44031 (local address 127.0.0.1:44148)
0928 12:21:43.777725 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:43.777736 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:43.777772 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:43.778030 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:43.778031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:43.778232 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:43.778383 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.778386 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.779048 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.779049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.779605 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.779607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.779921 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.779922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.780001 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.780370 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:43.780375 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:43.781709 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:43.783150 (+ 1441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:43.783152 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:43.783152 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:43.783338 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:43.783504 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:43.783505 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:43.783505 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:43.783563 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:43.783775 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:43.783776 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:43.783826 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 12:21:43.783885 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 12:21:43.783996 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 12:21:43.784102 9101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.777833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44148 (local address 127.2.56.60:44031)
0928 12:21:43.777868 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:43.777870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:43.777877 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:21:43.777912 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:43.777913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:43.777923 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:43.777966 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:43.778471 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.778975 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.779674 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.779865 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.779920 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.781786 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:43.781788 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:43.781788 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:43.781796 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:43.783077 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.783391 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.783392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.783393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.783449 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.783610 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.783611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.783612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.783725 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:43.783781 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:43.783911 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 12:21:43.783995 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 12:21:43.785305 8380 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.56.60:44031: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "35a79f2366a84381b917b76c334e55bc"
member_type: NON_VOTER
last_known_addr {
host: "127.2.56.60"
port: 44031
}
attrs {
promote: true
}
}
dest_uuid: "262a43ba51f54d5ba1fc806531019e58"
cas_config_opid_index: 5
I20260928 12:21:43.785609 8380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: NON_VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: true } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: NON_VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: true } }
I20260928 12:21:43.785969 8939 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 35a79f2366a84381b917b76c334e55bc (127.2.56.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: NON_VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: true } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: NON_VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: true } } }
I20260928 12:21:43.786877 8939 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.56.60:44031
I20260928 12:21:43.786944 8939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "262a43ba51f54d5ba1fc806531019e58" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: NON_VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: true } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: NON_VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: true } } }
I20260928 12:21:43.787019 8939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:43.787451 9031 master.cc:537] Master@127.2.56.60:44031 shutting down...
I20260928 12:21:43.787696 9073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:43.788785 9073 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [term 1 FOLLOWER]: Refusing update from remote peer 262a43ba51f54d5ba1fc806531019e58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 12:21:43.789259 8938 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: NON_VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 12:21:43.790375 8365 consensus_peers.cc:487] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 -> Peer 35a79f2366a84381b917b76c334e55bc (127.2.56.60:44031): Couldn't send request to peer 35a79f2366a84381b917b76c334e55bc. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 12:21:43.792769 8937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.787189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.61:42657 (local address 127.0.0.1:35308)
0928 12:21:43.787540 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:43.787552 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:43.787601 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:43.788995 (+ 1394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:43.788997 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:43.788999 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:43.789047 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.789048 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.791073 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.791074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.791929 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.791932 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.792521 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.792522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.792583 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.792612 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 12:21:43.792657 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 12:21:43.792698 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334}
I20260928 12:21:43.793021 9028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.787264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35308 (local address 127.2.56.61:42657)
0928 12:21:43.787310 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:43.787312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:43.787600 (+ 288us) server_negotiation.cc:407] Connection header received
0928 12:21:43.787680 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:43.787681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:43.787711 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:43.787753 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:43.790209 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.790916 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.792009 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.792452 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.792513 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.792604 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:43.792900 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 12:21:43.792951 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:21:43.793293 8987 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [term 1 LEARNER]: Refusing update from remote peer 262a43ba51f54d5ba1fc806531019e58: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 12:21:43.793357 9031 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:21:43.793445 9031 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:43.793497 9031 tablet_replica.cc:330] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc: stopping tablet replica
I20260928 12:21:43.793510 8938 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: NON_VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 12:21:43.794214 8987 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 35a79f2366a84381b917b76c334e55bc (127.2.56.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: NON_VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: true } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: NON_VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: true } } }
I20260928 12:21:43.795607 9126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "262a43ba51f54d5ba1fc806531019e58" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: NON_VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: true } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: NON_VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: true } } }
I20260928 12:21:43.795738 9016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 262a43ba51f54d5ba1fc806531019e58. Latest consensus state: current_term: 1 leader_uuid: "262a43ba51f54d5ba1fc806531019e58" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: NON_VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: true } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: NON_VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: true } } }
I20260928 12:21:43.795854 9016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [sys.catalog]: This master's current role is: LEARNER
I20260928 12:21:43.795751 9126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [sys.catalog]: This master's current role is: LEARNER
I20260928 12:21:43.803933 8938 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58: attempting to promote NON_VOTER 2b6ccb471f0e419c91dd33e886b9fe85 to VOTER
I20260928 12:21:43.804205 8938 consensus_queue.cc:237] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: false } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: NON_VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: true } }
I20260928 12:21:43.804714 8937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.804599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:44031 (local address 127.0.0.1:44160)
0928 12:21:43.804642 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:43.804683 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.60:44031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 12:21:43.804986 8987 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [term 1 LEARNER]: Refusing update from remote peer 262a43ba51f54d5ba1fc806531019e58: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 12:21:43.805065 8937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.804992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:44031 (local address 127.0.0.1:44168)
0928 12:21:43.805024 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:43.805045 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.60:44031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260928 12:21:43.805115 8365 consensus_peers.cc:487] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 -> Peer 35a79f2366a84381b917b76c334e55bc (127.2.56.60:44031): Couldn't send request to peer 35a79f2366a84381b917b76c334e55bc. Status: Network error: Client connection negotiation failed: client connection to 127.2.56.60:44031: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 12:21:43.805226 9127 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 12:21:43.805806 8938 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2b6ccb471f0e419c91dd33e886b9fe85 (127.2.56.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: false } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: NON_VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: true } } }
I20260928 12:21:43.806952 9031 master.cc:552] Master@127.2.56.60:44031 shutdown complete.
I20260928 12:21:43.806998 9031 master_runner.cc:305] Clearing existing system tablet
I20260928 12:21:43.807129 9031 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:43.807171 9031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:43.807340 8938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "262a43ba51f54d5ba1fc806531019e58" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: false } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: NON_VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: true } } }
I20260928 12:21:43.807425 8938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:43.807473 8938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "262a43ba51f54d5ba1fc806531019e58" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: false } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: NON_VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: true } } }
I20260928 12:21:43.807511 8938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:43.807823 8987 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2b6ccb471f0e419c91dd33e886b9fe85 (127.2.56.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: false } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: NON_VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: true } } }
I20260928 12:21:43.808557 9126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "262a43ba51f54d5ba1fc806531019e58" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: false } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: NON_VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: true } } }
I20260928 12:21:43.808638 9126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:21:43.809458 9031 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 12:21:43.809902 9129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:43.810014 9031 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:43.810066 9031 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:43.810091 9031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-2/wal
uuid: "35a79f2366a84381b917b76c334e55bc"
format_stamp: "Formatted at 2026-09-28 12:21:43 on dist-test-slave-920p"
server_key: "1994258b87284b10bef20b9c673923f4"
server_key_iv: "3d36bb5581b8c7002f71dbb3cefac854"
server_key_version: "encryptionkey@0"
I20260928 12:21:43.810743 9031 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:43.811990 9031 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 12:21:43.812048 9031 log.cc:1196] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 12:21:43.812228 9031 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc: Deleting consensus metadata
I20260928 12:21:43.812338 9031 master_runner.cc:315] Copying system tablet from 127.2.56.62:43477
I20260928 12:21:43.812902 9031 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc: tablet copy: Beginning tablet copy session from remote peer at address 127.2.56.62:43477
I20260928 12:21:43.818663 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.813238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51746 (local address 127.2.56.62:43477)
0928 12:21:43.813404 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:43.813407 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:43.813414 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:21:43.813438 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:43.813438 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:43.813449 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:43.813503 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:43.813817 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.814352 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.815056 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.815265 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.815318 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.816251 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:43.816253 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:43.816253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:43.816261 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:43.817563 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.817861 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.817862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.817863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.817929 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.818140 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.818141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.818142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.818259 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:43.818317 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:43.818479 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 12:21:43.818566 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:21:43.818677 9135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.813180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.0.0.1:51746)
0928 12:21:43.813321 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:43.813331 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:43.813384 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:43.813509 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:43.813509 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:43.813699 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:43.813751 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.813753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.814429 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.814430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.814986 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.814988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.815323 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.815324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.815389 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.815847 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:43.815852 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:43.816170 (+ 318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:43.817634 (+ 1464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:43.817635 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:43.817636 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:43.817803 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:43.818030 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:43.818031 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:43.818031 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:43.818087 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:43.818313 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:43.818314 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:43.818381 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 12:21:43.818444 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 12:21:43.818567 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":67,"threads_started":1}
I20260928 12:21:43.819193 8400 tablet_copy_service.cc:140] P 262a43ba51f54d5ba1fc806531019e58: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 35a79f2366a84381b917b76c334e55bc ({username='oryx', principal='oryx/127.2.56.60@KRBTEST.COM'} at 127.0.0.1:51746)
I20260928 12:21:43.819253 8400 tablet_copy_service.cc:161] P 262a43ba51f54d5ba1fc806531019e58: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 35a79f2366a84381b917b76c334e55bc at {username='oryx', principal='oryx/127.2.56.60@KRBTEST.COM'} at 127.0.0.1:51746: session id = 35a79f2366a84381b917b76c334e55bc-00000000000000000000000000000000
I20260928 12:21:43.819772 8400 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 12:21:43.820207 9031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 12:21:43.821162 9031 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc: tablet copy: Starting download of 0 data blocks...
I20260928 12:21:43.821348 9031 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc: tablet copy: Starting download of 1 WAL segments...
I20260928 12:21:43.821977 9031 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 12:21:43.822736 8400 tablet_copy_service.cc:342] P 262a43ba51f54d5ba1fc806531019e58: Request end of tablet copy session 35a79f2366a84381b917b76c334e55bc-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.56.60@KRBTEST.COM'} at 127.0.0.1:51746
I20260928 12:21:43.822788 8400 tablet_copy_service.cc:433] P 262a43ba51f54d5ba1fc806531019e58: ending tablet copy session 35a79f2366a84381b917b76c334e55bc-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 35a79f2366a84381b917b76c334e55bc
I20260928 12:21:43.825480 9031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:43.826659 9136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:43.826692 9139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:43.829424 9137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:43.833456 9031 server_base.cc:1034] running on GCE node
I20260928 12:21:43.833603 9031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:43.833760 9031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:43.834872 9031 hybrid_clock.cc:648] HybridClock initialized: now 1790598103834864 us; error 32 us; skew 500 ppm
Sep 28 12:21:43 dist-test-slave-920p krb5kdc[8335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598103, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:43.836261 9031 init.cc:377] Logged in from keytab as oryx/127.2.56.60@KRBTEST.COM (short username oryx)
I20260928 12:21:43.836550 9031 webserver.cc:460] Webserver started at http://127.2.56.60:35455/ using document root <none> and password file <none>
I20260928 12:21:43.836668 9031 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:43.836701 9031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:43.837378 9031 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:43.837899 9146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:43.838027 9031 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:43.838069 9031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-2/wal
uuid: "35a79f2366a84381b917b76c334e55bc"
format_stamp: "Formatted at 2026-09-28 12:21:43 on dist-test-slave-920p"
server_key: "1994258b87284b10bef20b9c673923f4"
server_key_iv: "3d36bb5581b8c7002f71dbb3cefac854"
server_key_version: "encryptionkey@0"
I20260928 12:21:43.838136 9031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:43.856958 9031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:43.858611 9031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:43.862134 9031 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.60:44031
I20260928 12:21:43.862479 9031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/master-2/data/info.pb
I20260928 12:21:43.862736 9198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.60:44031 every 8 connection(s)
I20260928 12:21:43.863422 9199 sys_catalog.cc:263] Verifying existing consensus state
I20260928 12:21:43.863814 9199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc: Bootstrap starting.
I20260928 12:21:43.866356 9199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:21:43.866564 9199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc: Bootstrap complete.
I20260928 12:21:43.866788 9199 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: false } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: NON_VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: true } }
I20260928 12:21:43.866869 9199 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 35a79f2366a84381b917b76c334e55bc, State: Initialized, Role: LEARNER
I20260928 12:21:43.866902 9199 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: false } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: NON_VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: true } }
I20260928 12:21:43.867116 9199 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:21:43.867116 9202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: false } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: NON_VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: true } } }
I20260928 12:21:43.867249 9202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [sys.catalog]: This master's current role is: LEARNER
I20260928 12:21:43.868572 9213 catalog_manager.cc:1259] Loaded cluster ID: f5f2d04c9c0b48a1b4ff09425dfdc0bd
I20260928 12:21:43.868620 9213 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc: loading cluster ID for follower catalog manager: success
I20260928 12:21:43.869707 9213 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc: acquiring CA information for follower catalog manager: success
I20260928 12:21:43.870003 9213 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 12:21:43.908981 9124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:43.909338 9124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:43.909379 9124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:43.911203 9124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:43.911273 9124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:43.911311 9124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.1
I20260928 12:21:43.913435 9124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:43591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.1.keytab
--principal=oryx/127.2.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.1:42075
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.56.1
--webserver_port=45751
--tserver_master_addrs=127.2.56.62:43477,127.2.56.61:42657,127.2.56.60:44031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.1
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:43.913726 9124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:43.914059 9124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:43.919844 9218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:43.919844 9221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:43.920053 9124 server_base.cc:1034] running on GCE node
W20260928 12:21:43.919909 9219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:43.920395 9124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:43.920634 9124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:43.921772 9124 hybrid_clock.cc:648] HybridClock initialized: now 1790598103921755 us; error 33 us; skew 500 ppm
Sep 28 12:21:43 dist-test-slave-920p krb5kdc[8335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598103, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:43.924571 9124 init.cc:377] Logged in from keytab as oryx/127.2.56.1@KRBTEST.COM (short username oryx)
I20260928 12:21:43.925035 9124 webserver.cc:460] Webserver started at http://127.2.56.1:45751/ using document root <none> and password file <none>
I20260928 12:21:43.925289 9124 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:43.925365 9124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:43.926839 9124 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:21:43.927386 9228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:43.927556 9124 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:43.927640 9124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-0/wal
uuid: "53c647c83fb847d090c0cc0c1e3e6f60"
format_stamp: "Formatted at 2026-09-28 12:21:42 on dist-test-slave-920p"
server_key: "894bf78498255acfad95a7a45c8e207c"
server_key_iv: "3415d5e41690eaeb9b3d5b5d827d272d"
server_key_version: "encryptionkey@0"
I20260928 12:21:43.928037 9124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:43.939016 9124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:43.940929 9124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:43.941066 9124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:43.941324 9124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:21:43.941674 9124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:43.941708 9124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:43.941740 9124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:43.941762 9124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:43.948144 9124 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.1:42075
I20260928 12:21:43.948210 9341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.1:42075 every 8 connection(s)
I20260928 12:21:43.948580 9124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
I20260928 12:21:43.953019 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 9124
I20260928 12:21:43.953193 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 8576
Sep 28 12:21:43 dist-test-slave-920p krb5kdc[8335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598103, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.60@KRBTEST.COM
Sep 28 12:21:43 dist-test-slave-920p krb5kdc[8335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598103, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM
Sep 28 12:21:43 dist-test-slave-920p krb5kdc[8335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598103, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.61@KRBTEST.COM
I20260928 12:21:43.959806 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.2:34785
--local_ip_for_outbound_sockets=127.2.56.2
--tserver_master_addrs=127.2.56.62:43477
--webserver_port=40811
--webserver_interface=127.2.56.2
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.2.keytab
--principal=oryx/127.2.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:43591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.56.62:43477,127.2.56.61:42657
--tserver_master_addrs=127.2.56.62:43477,127.2.56.61:42657,127.2.56.60:44031 with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:21:43.961220 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.949703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:52935 (local address 127.2.56.62:43477)
0928 12:21:43.949784 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:43.949786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:43.949795 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:21:43.950520 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:43.950522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:43.950537 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:43.950580 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:43.952218 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.952711 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.954281 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.954588 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.954627 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.958312 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:43.958314 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:43.958314 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:43.958322 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:43.959738 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.960210 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.960211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.960212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.960270 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.960442 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.960443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.960443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.960591 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:43.960656 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:43.961030 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 12:21:43.961121 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 12:21:43.961377 9348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.949488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:59193 (local address 127.2.56.60:44031)
0928 12:21:43.949848 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:43.949851 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:43.950004 (+ 153us) server_negotiation.cc:407] Connection header received
0928 12:21:43.950524 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:43.950525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:43.950540 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:43.950608 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:43.952801 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.953791 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.954762 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.955055 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.955117 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.957857 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:43.957859 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:43.957860 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:43.957869 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:43.959595 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.960099 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.960101 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.960101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.960164 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.960440 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.960441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.960441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.960781 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:43.960836 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:43.961173 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 12:21:43.961285 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":65,"threads_started":1}
I20260928 12:21:43.962015 9346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.949427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.2.56.1:52935)
0928 12:21:43.949631 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:43.949683 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:43.950344 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:43.950651 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:43.950652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:43.951182 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:43.951470 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.951476 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.952786 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.952787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.954200 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.954201 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.954703 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.954704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.954757 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.955470 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:43.955474 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:43.958201 (+ 2727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:43.959936 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:43.959936 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:43.959937 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:43.960133 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:43.960332 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:43.960332 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:43.960333 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:43.960377 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:43.960644 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:43.960651 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:43.960840 (+ 189us) client_negotiation.cc:769] Sending connection context
0928 12:21:43.961330 (+ 490us) client_negotiation.cc:240] Negotiation successful
0928 12:21:43.961770 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":339,"thread_start_us":99,"threads_started":1}
I20260928 12:21:43.962025 9349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.949428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:44031 (local address 127.2.56.1:59193)
0928 12:21:43.949930 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:43.949941 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:43.950346 (+ 405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:43.950934 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:43.950935 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:43.951193 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:43.951469 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.951475 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.953884 (+ 2409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.953885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.954674 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.954676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.955120 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.955121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.955168 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.955587 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:43.955591 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:43.957745 (+ 2154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:43.959685 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:43.959695 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:43.959699 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:43.960023 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:43.960227 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:43.960228 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:43.960228 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:43.960366 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:43.960860 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:43.960861 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:43.960894 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 12:21:43.960969 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 12:21:43.961712 (+ 743us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":385,"thread_start_us":82,"threads_started":1}
I20260928 12:21:43.962713 9342 heartbeater.cc:347] Connected to a master server at 127.2.56.60:44031
I20260928 12:21:43.962764 9028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.949706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:34929 (local address 127.2.56.61:42657)
0928 12:21:43.949758 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:43.949761 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:43.949775 (+ 14us) server_negotiation.cc:407] Connection header received
0928 12:21:43.950463 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:43.950465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:43.950479 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:43.950531 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:43.951577 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.952127 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.954124 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.954423 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.954465 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.958462 (+ 3997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:43.958463 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:43.958464 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:43.958474 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:43.960958 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.961777 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.961778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.961779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.961843 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:43.962026 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:43.962027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:43.962027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:43.962172 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:43.962225 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:43.962589 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 12:21:43.962678 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 12:21:43.962829 9342 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:43.963044 9342 heartbeater.cc:511] Master 127.2.56.60:44031 requested a full tablet report, sending...
I20260928 12:21:43.963132 9347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:43.949427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.61:42657 (local address 127.2.56.1:34929)
0928 12:21:43.949631 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:43.949683 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:43.950342 (+ 659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:43.950628 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:43.950632 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:43.951193 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:43.951469 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.951476 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.952217 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.952218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.954030 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:43.954033 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:43.954562 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:43.954563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:43.954715 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:43.955420 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:43.955432 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:43.958334 (+ 2902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:43.961069 (+ 2735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:43.961070 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:43.961070 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:43.961688 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:43.961908 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:43.961908 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:43.961909 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:43.961962 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:43.962426 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:43.962427 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:43.962478 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 12:21:43.962885 (+ 407us) client_negotiation.cc:240] Negotiation successful
0928 12:21:43.962991 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":52,"thread_start_us":91,"threads_started":1}
I20260928 12:21:43.963614 9163 ts_manager.cc:194] Registered new tserver with Master: 53c647c83fb847d090c0cc0c1e3e6f60 (127.2.56.1:42075)
I20260928 12:21:43.963769 9344 heartbeater.cc:347] Connected to a master server at 127.2.56.61:42657
I20260928 12:21:43.963814 9344 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:43.963879 9344 heartbeater.cc:511] Master 127.2.56.61:42657 requested a full tablet report, sending...
I20260928 12:21:43.964013 9343 heartbeater.cc:347] Connected to a master server at 127.2.56.62:43477
I20260928 12:21:43.964046 9343 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:43.964093 9343 heartbeater.cc:511] Master 127.2.56.62:43477 requested a full tablet report, sending...
I20260928 12:21:43.964274 8971 ts_manager.cc:194] Registered new tserver with Master: 53c647c83fb847d090c0cc0c1e3e6f60 (127.2.56.1:42075)
I20260928 12:21:43.964372 8380 ts_manager.cc:194] Re-registered known tserver with Master: 53c647c83fb847d090c0cc0c1e3e6f60 (127.2.56.1:42075)
I20260928 12:21:43.964792 8380 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.56.1@KRBTEST.COM'} at 127.2.56.1:52935
W20260928 12:21:44.050817 9350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:44.051059 9350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:44.051090 9350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:44.052282 9350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:44.052320 9350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:44.052352 9350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.2
I20260928 12:21:44.053867 9350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:43591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.2.keytab
--principal=oryx/127.2.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.2:34785
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.56.2
--webserver_port=40811
--tserver_master_addrs=127.2.56.62:43477,127.2.56.61:42657,127.2.56.60:44031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.2
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:44.054126 9350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:44.054350 9350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:44.059504 9358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:44.059475 9356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:44.059471 9355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:44.059592 9350 server_base.cc:1034] running on GCE node
I20260928 12:21:44.059960 9350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:44.060204 9350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:44.061312 9350 hybrid_clock.cc:648] HybridClock initialized: now 1790598104061300 us; error 29 us; skew 500 ppm
Sep 28 12:21:44 dist-test-slave-920p krb5kdc[8335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598104, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:44.063930 9350 init.cc:377] Logged in from keytab as oryx/127.2.56.2@KRBTEST.COM (short username oryx)
I20260928 12:21:44.064383 9350 webserver.cc:460] Webserver started at http://127.2.56.2:40811/ using document root <none> and password file <none>
I20260928 12:21:44.064579 9350 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:44.064630 9350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:44.066025 9350 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260928 12:21:44.066561 9365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:44.066721 9350 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:44.066804 9350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-1/wal
uuid: "9c6a797c45334b15bca72df411c5d1bb"
format_stamp: "Formatted at 2026-09-28 12:21:43 on dist-test-slave-920p"
server_key: "5367e319c4e4b609edcd78e801d4ad16"
server_key_iv: "e07e542192ebaf1b339d88a6fd4291e1"
server_key_version: "encryptionkey@0"
I20260928 12:21:44.067245 9350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:44.091835 9350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:44.093843 9350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:44.093968 9350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:44.094194 9350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:21:44.094499 9350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:44.094529 9350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:44.094552 9350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:44.094566 9350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:44.100978 9350 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.2:34785
I20260928 12:21:44.101050 9478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.2:34785 every 8 connection(s)
I20260928 12:21:44.101435 9350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
I20260928 12:21:44.106431 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 9350
I20260928 12:21:44.106603 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 8710
Sep 28 12:21:44 dist-test-slave-920p krb5kdc[8335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598104, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.60@KRBTEST.COM
Sep 28 12:21:44 dist-test-slave-920p krb5kdc[8335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598104, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.61@KRBTEST.COM
Sep 28 12:21:44 dist-test-slave-920p krb5kdc[8335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598104, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM
I20260928 12:21:44.113487 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:44.102473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:47441 (local address 127.2.56.62:43477)
0928 12:21:44.102524 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:44.102526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:44.102535 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:21:44.103629 (+ 1094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:44.103631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:44.103645 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:44.103697 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:44.105438 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.105936 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.108055 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.108333 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.108374 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:44.110478 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:44.110480 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:44.110480 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:44.110488 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:44.111979 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:44.112455 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:44.112456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:44.112457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:44.112513 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:44.112758 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:44.112758 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:44.112759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:44.112880 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:44.112920 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:44.113294 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 12:21:44.113375 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 12:21:44.113487 9348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:44.102263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:39735 (local address 127.2.56.60:44031)
0928 12:21:44.102324 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:44.102327 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:44.102750 (+ 423us) server_negotiation.cc:407] Connection header received
0928 12:21:44.103656 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:44.103658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:44.103675 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:44.103705 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:44.104486 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.105338 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.106397 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.106727 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.106793 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:44.110112 (+ 3319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:44.110114 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:44.110115 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:44.110124 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:44.112355 (+ 2231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:44.112674 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:44.112675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:44.112676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:44.112726 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:44.112900 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:44.112901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:44.112901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:44.113037 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:44.113074 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:44.113303 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 12:21:44.113389 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 12:21:44.113488 9028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:44.102472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:53969 (local address 127.2.56.61:42657)
0928 12:21:44.102525 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:44.102527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:44.102537 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:21:44.103657 (+ 1120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:44.103659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:44.103673 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:44.103705 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:44.104508 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.105349 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.107274 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.107551 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.107595 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:44.110455 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:44.110456 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:44.110456 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:44.110464 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:44.112151 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:44.112545 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:44.112546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:44.112546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:44.112602 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:44.112788 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:44.112789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:44.112789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:44.112914 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:44.112954 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:44.113303 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 12:21:44.113387 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 12:21:44.114030 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:44.102197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:44031 (local address 127.2.56.2:39735)
0928 12:21:44.102675 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:44.102684 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:44.103544 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:44.103774 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:44.103778 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:44.104202 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:44.104390 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:44.104396 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.105440 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.105441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:44.106317 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:44.106319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.106862 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.106862 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:44.106998 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:44.107558 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:44.107570 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:44.110003 (+ 2433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:44.112432 (+ 2429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:44.112433 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:44.112433 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:44.112601 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:44.112789 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:44.112790 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:44.112790 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:44.112838 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:44.113116 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:44.113117 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:44.113162 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 12:21:44.113638 (+ 476us) client_negotiation.cc:240] Negotiation successful
0928 12:21:44.113821 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":438,"thread_start_us":81,"threads_started":1}
I20260928 12:21:44.114032 9483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:44.102197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.61:42657 (local address 127.2.56.2:53969)
0928 12:21:44.102370 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:44.102418 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:44.103533 (+ 1115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:44.103774 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:44.103778 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:44.104191 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:44.104390 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:44.104396 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.106403 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.106404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:44.107177 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:44.107179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.107675 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.107676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:44.107738 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:44.108076 (+ 338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:44.108081 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:44.110349 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:44.112241 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:44.112242 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:44.112242 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:44.112469 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:44.112667 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:44.112668 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:44.112668 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:44.112721 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:44.112996 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:44.112997 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:44.113133 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 12:21:44.113579 (+ 446us) client_negotiation.cc:240] Negotiation successful
0928 12:21:44.113765 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":2,"thread_start_us":87,"threads_started":1}
I20260928 12:21:44.114229 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.3:43087
--local_ip_for_outbound_sockets=127.2.56.3
--tserver_master_addrs=127.2.56.62:43477
--webserver_port=38823
--webserver_interface=127.2.56.3
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.3.keytab
--principal=oryx/127.2.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:43591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.56.62:43477,127.2.56.61:42657
--tserver_master_addrs=127.2.56.62:43477,127.2.56.61:42657,127.2.56.60:44031 with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:21:44.114544 9481 heartbeater.cc:347] Connected to a master server at 127.2.56.61:42657
I20260928 12:21:44.114653 9481 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:44.114843 9481 heartbeater.cc:511] Master 127.2.56.61:42657 requested a full tablet report, sending...
I20260928 12:21:44.114852 9484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:44.102197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.2.56.2:47441)
0928 12:21:44.102375 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:44.102424 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:44.103538 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:44.103773 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:44.103777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:44.104175 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:44.104389 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:44.104394 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.106004 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.106005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:44.107763 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:44.107766 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.108462 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.108463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:44.108524 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:44.108920 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:44.108925 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:44.110377 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:44.112064 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:44.112070 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:44.112073 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:44.112368 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:44.112576 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:44.112576 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:44.112577 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:44.112695 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:44.112960 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:44.112966 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:44.113131 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 12:21:44.114006 (+ 875us) client_negotiation.cc:240] Negotiation successful
0928 12:21:44.114695 (+ 689us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":2,"thread_start_us":94,"threads_started":1}
I20260928 12:21:44.115371 8971 ts_manager.cc:194] Registered new tserver with Master: 9c6a797c45334b15bca72df411c5d1bb (127.2.56.2:34785)
I20260928 12:21:44.115839 9479 heartbeater.cc:347] Connected to a master server at 127.2.56.60:44031
I20260928 12:21:44.115895 9479 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:44.115959 9479 heartbeater.cc:511] Master 127.2.56.60:44031 requested a full tablet report, sending...
I20260928 12:21:44.116243 9163 ts_manager.cc:194] Registered new tserver with Master: 9c6a797c45334b15bca72df411c5d1bb (127.2.56.2:34785)
I20260928 12:21:44.116406 9480 heartbeater.cc:347] Connected to a master server at 127.2.56.62:43477
I20260928 12:21:44.116448 9480 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:44.116513 9480 heartbeater.cc:511] Master 127.2.56.62:43477 requested a full tablet report, sending...
I20260928 12:21:44.116751 8380 ts_manager.cc:194] Re-registered known tserver with Master: 9c6a797c45334b15bca72df411c5d1bb (127.2.56.2:34785)
I20260928 12:21:44.117143 8380 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.56.2@KRBTEST.COM'} at 127.2.56.2:47441
W20260928 12:21:44.202306 9486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:44.202543 9486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:44.202565 9486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:44.203744 9486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:44.203790 9486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:44.203820 9486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.3
I20260928 12:21:44.205341 9486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:43591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.3.keytab
--principal=oryx/127.2.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.3:43087
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.56.3
--webserver_port=38823
--tserver_master_addrs=127.2.56.62:43477,127.2.56.61:42657,127.2.56.60:44031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.3
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:44.205586 9486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:44.205822 9486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:44.211145 9491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:44.211143 9492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:44.211216 9486 server_base.cc:1034] running on GCE node
W20260928 12:21:44.211160 9494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:44.211583 9486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:44.211819 9486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:44.212955 9486 hybrid_clock.cc:648] HybridClock initialized: now 1790598104212948 us; error 42 us; skew 500 ppm
Sep 28 12:21:44 dist-test-slave-920p krb5kdc[8335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598104, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:44.215651 9486 init.cc:377] Logged in from keytab as oryx/127.2.56.3@KRBTEST.COM (short username oryx)
I20260928 12:21:44.216125 9486 webserver.cc:460] Webserver started at http://127.2.56.3:38823/ using document root <none> and password file <none>
I20260928 12:21:44.216368 9486 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:44.216511 9486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:44.218052 9486 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:21:44.218634 9501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:44.218833 9486 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:44.218940 9486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-2/wal
uuid: "cf83a5232738461e9eace3fc21dd0929"
format_stamp: "Formatted at 2026-09-28 12:21:43 on dist-test-slave-920p"
server_key: "d19d7a7aef0e6b9a4cf24b50f7cb56d4"
server_key_iv: "e089f4805cf47704530b7f7c65c4504c"
server_key_version: "encryptionkey@0"
I20260928 12:21:44.219381 9486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:44.228255 9486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:44.230268 9486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:44.230382 9486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:44.230568 9486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:21:44.230876 9486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:44.230908 9486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:44.230947 9486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:44.230970 9486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:44.237192 9486 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.3:43087
I20260928 12:21:44.237293 9614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.3:43087 every 8 connection(s)
I20260928 12:21:44.237654 9486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
I20260928 12:21:44.239549 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 9486
Sep 28 12:21:44 dist-test-slave-920p krb5kdc[8335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598104, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.61@KRBTEST.COM
Sep 28 12:21:44 dist-test-slave-920p krb5kdc[8335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598104, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM
Sep 28 12:21:44 dist-test-slave-920p krb5kdc[8335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598104, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.60@KRBTEST.COM
I20260928 12:21:44.251639 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:44.241444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35316 (local address 127.2.56.61:42657)
0928 12:21:44.241897 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:44.241899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:44.242036 (+ 137us) server_negotiation.cc:407] Connection header received
0928 12:21:44.242058 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:44.242058 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:44.242070 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:44.242123 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:44.242872 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.243597 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.244708 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.245056 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.245115 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:44.246967 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:44.246968 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:44.246968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:44.246975 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:44.250188 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:44.250553 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:44.250554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:44.250555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:44.250608 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:44.250792 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:44.250793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:44.250793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:44.250927 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:44.251018 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:44.251447 (+ 429us) server_negotiation.cc:299] Negotiation successful
0928 12:21:44.251522 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":59,"threads_started":1}
I20260928 12:21:44.252190 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:44.240923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44172 (local address 127.2.56.60:44031)
0928 12:21:44.241698 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:44.241700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:44.243826 (+ 2126us) server_negotiation.cc:407] Connection header received
0928 12:21:44.244752 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:44.244753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:44.244766 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:44.244819 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:44.245505 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.246251 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.247246 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.247534 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.247588 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:44.249048 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:44.249049 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:44.249050 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:44.249056 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:44.250875 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:44.251251 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:44.251252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:44.251253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:44.251307 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:44.251489 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:44.251490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:44.251490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:44.251614 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:44.251701 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:44.252013 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 12:21:44.252091 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":755,"thread_start_us":60,"threads_started":1}
I20260928 12:21:44.252746 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:44.238393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:45439 (local address 127.2.56.62:43477)
0928 12:21:44.238458 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:44.238461 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:44.238925 (+ 464us) server_negotiation.cc:407] Connection header received
0928 12:21:44.239479 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:44.239480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:44.239497 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:44.239553 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:44.240617 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.241382 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.243021 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.243322 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.243361 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:44.248401 (+ 5040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:44.248403 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:44.248403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:44.248412 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:44.251061 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:44.251505 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:44.251506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:44.251507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:44.251559 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:44.252056 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:44.252057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:44.252057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:44.252188 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:44.252228 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:44.252576 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 12:21:44.252653 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 12:21:44.252974 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:44.238492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.2.56.3:45439)
0928 12:21:44.238844 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:44.238854 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:44.239372 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:44.239649 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:44.239653 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:44.240171 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:44.240503 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:44.240510 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.241633 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.241633 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:44.242935 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:44.242938 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.243450 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.243450 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:44.243598 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:44.244288 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:44.244299 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:44.248096 (+ 3797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:44.251143 (+ 3047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:44.251144 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:44.251144 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:44.251425 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:44.251851 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:44.251852 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:44.251852 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:44.251980 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:44.252274 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:44.252275 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:44.252444 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 12:21:44.252552 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 12:21:44.252740 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":112,"thread_start_us":85,"threads_started":1}
I20260928 12:21:44.252981 9348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:44.238492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:33319 (local address 127.2.56.60:44031)
0928 12:21:44.238618 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:44.238620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:44.238913 (+ 293us) server_negotiation.cc:407] Connection header received
0928 12:21:44.239625 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:44.239627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:44.239639 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:44.239690 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:44.240609 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.241453 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.243030 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.243355 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.243389 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:44.249167 (+ 5778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:44.249168 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:44.249168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:44.249175 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:44.250968 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:44.251718 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:44.251719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:44.251720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:44.251775 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:44.252057 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:44.252058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:44.252058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:44.252174 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:44.252213 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:44.252813 (+ 600us) server_negotiation.cc:299] Negotiation successful
0928 12:21:44.252888 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260928 12:21:44.253512 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:44.238404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:44031 (local address 127.2.56.3:33319)
0928 12:21:44.238771 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:44.238819 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:44.239380 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:44.239742 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:44.239743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:44.240207 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:44.240503 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:44.240510 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.241651 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.241652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:44.242931 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:44.242934 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.243455 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.243456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:44.243598 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:44.244504 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:44.244508 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:44.249079 (+ 4571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:44.251039 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:44.251047 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:44.251050 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:44.251638 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:44.251835 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:44.251835 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:44.251836 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:44.251978 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:44.252254 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:44.252260 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:44.252690 (+ 430us) client_negotiation.cc:769] Sending connection context
0928 12:21:44.253085 (+ 395us) client_negotiation.cc:240] Negotiation successful
0928 12:21:44.253422 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":81,"thread_start_us":109,"threads_started":1}
I20260928 12:21:44.254115 9615 heartbeater.cc:347] Connected to a master server at 127.2.56.60:44031
I20260928 12:21:44.254217 9615 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:44.254424 9615 heartbeater.cc:511] Master 127.2.56.60:44031 requested a full tablet report, sending...
I20260928 12:21:44.254601 9028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:44.238491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:39739 (local address 127.2.56.61:42657)
0928 12:21:44.238577 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:44.238579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:44.239820 (+ 1241us) server_negotiation.cc:407] Connection header received
0928 12:21:44.239916 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:44.239917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:44.239928 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:44.239956 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:44.240609 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.241430 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.243030 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.243324 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.243363 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:44.248380 (+ 5017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:44.248381 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:44.248381 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:44.248388 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:44.249784 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:44.253836 (+ 4052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:44.253837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:44.253838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:44.253899 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:44.254092 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:44.254093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:44.254094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:44.254212 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:44.254265 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:44.254448 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 12:21:44.254520 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 12:21:44.254601 9616 heartbeater.cc:347] Connected to a master server at 127.2.56.62:43477
I20260928 12:21:44.254644 9616 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:44.254706 9616 heartbeater.cc:511] Master 127.2.56.62:43477 requested a full tablet report, sending...
I20260928 12:21:44.254878 9163 ts_manager.cc:194] Registered new tserver with Master: cf83a5232738461e9eace3fc21dd0929 (127.2.56.3:43087)
I20260928 12:21:44.254917 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:44.238404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.61:42657 (local address 127.2.56.3:39739)
0928 12:21:44.239755 (+ 1351us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:44.239765 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:44.239845 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:44.240021 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:44.240022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:44.240233 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:44.240503 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:44.240510 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.241560 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.241561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:44.242931 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:44.242934 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.243453 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.243454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:44.243598 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:44.244406 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:44.244410 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:44.248266 (+ 3856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:44.252703 (+ 4437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:44.252704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:44.252704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:44.253709 (+ 1005us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:44.253970 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:44.253970 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:44.253971 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:44.254026 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:44.254310 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:44.254310 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:44.254350 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 12:21:44.254680 (+ 330us) client_negotiation.cc:240] Negotiation successful
0928 12:21:44.254821 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1328,"mutex_wait_us":415,"thread_start_us":71,"threads_started":1}
I20260928 12:21:44.254947 8380 ts_manager.cc:194] Re-registered known tserver with Master: cf83a5232738461e9eace3fc21dd0929 (127.2.56.3:43087)
I20260928 12:21:44.255151 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:44.241491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51758 (local address 127.2.56.62:43477)
0928 12:21:44.243894 (+ 2403us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:44.243895 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:44.243901 (+ 6us) server_negotiation.cc:407] Connection header received
0928 12:21:44.243933 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:44.243933 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:44.243945 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:44.244012 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:44.245191 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.245826 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.246771 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.247060 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.247114 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:44.249437 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:44.249438 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:44.249438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:44.249446 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:44.253868 (+ 4422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:44.254204 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:44.254205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:44.254206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:44.254257 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:44.254409 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:44.254409 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:44.254410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:44.254521 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:44.254601 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:44.254989 (+ 388us) server_negotiation.cc:299] Negotiation successful
0928 12:21:44.255058 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2374,"thread_start_us":77,"threads_started":1}
I20260928 12:21:44.255388 9617 heartbeater.cc:347] Connected to a master server at 127.2.56.61:42657
I20260928 12:21:44.255422 9617 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:44.255398 8380 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.56.3@KRBTEST.COM'} at 127.2.56.3:45439
I20260928 12:21:44.255466 9617 heartbeater.cc:511] Master 127.2.56.61:42657 requested a full tablet report, sending...
I20260928 12:21:44.255734 8971 ts_manager.cc:194] Registered new tserver with Master: cf83a5232738461e9eace3fc21dd0929 (127.2.56.3:43087)
I20260928 12:21:44.256884 8380 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51758:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:21:44.258718 8380 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:21:44.265761 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:44.262336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:43087 (local address 127.0.0.1:54116)
0928 12:21:44.262576 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:44.262585 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:44.262632 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:44.262885 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:44.262885 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:44.262887 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:44.262960 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:44.262961 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.263840 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.263841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:44.264608 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:44.264610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.265514 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.265515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:44.265589 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:44.265594 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:21:44.265648 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 12:21:44.265687 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":80,"threads_started":1}
I20260928 12:21:44.265769 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:44.262430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54116 (local address 127.2.56.3:43087)
0928 12:21:44.262735 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:44.262739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:44.262747 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:21:44.262783 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:44.262786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:44.262805 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:44.262879 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:44.263030 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.263775 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.264686 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.265401 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.265487 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:44.265566 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:44.265662 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 12:21:44.265693 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":71,"threads_started":1}
I20260928 12:21:44.266806 8937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:44.262318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:42075 (local address 127.0.0.1:38852)
0928 12:21:44.262537 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:44.262547 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:44.262588 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:44.262818 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:44.262819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:44.262821 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:44.262869 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:44.262871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.264865 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.264866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:44.265776 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:44.265778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.266596 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.266597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:44.266655 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:44.266658 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:21:44.266708 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 12:21:44.266746 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":19}
I20260928 12:21:44.266848 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:44.262464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38852 (local address 127.2.56.1:42075)
0928 12:21:44.262634 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:44.262640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:44.262650 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:21:44.262695 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:44.262697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:44.262715 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:44.262939 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:44.263018 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.264769 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.265860 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.266520 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.266576 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:44.266644 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:44.266723 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 12:21:44.266772 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":84,"threads_started":1}
I20260928 12:21:44.266780 9549 tablet_service.cc:1467] Processing CreateTablet for tablet 19c822da2082490d94637fdaae448d20 (DEFAULT_TABLE table=test-table [id=82e5a42e465d4304b4ce03d45e0a3035]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:44.267207 9549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19c822da2082490d94637fdaae448d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:44.267505 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:44.262741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51708 (local address 127.2.56.2:34785)
0928 12:21:44.262934 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:44.262937 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:44.263052 (+ 115us) server_negotiation.cc:407] Connection header received
0928 12:21:44.263112 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:44.263114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:44.263132 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:44.263209 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:44.263317 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.264759 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.266139 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.267174 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.267237 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:44.267312 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:44.267387 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 12:21:44.267426 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":79,"threads_started":1}
I20260928 12:21:44.267544 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:44.262411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:34785 (local address 127.0.0.1:51708)
0928 12:21:44.262983 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:44.262992 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:44.263048 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:44.263212 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:44.263213 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:44.263215 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:44.263259 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:44.263260 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.264837 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.264838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:44.266057 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:44.266061 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.267263 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.267264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:44.267321 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:44.267326 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:21:44.267375 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 12:21:44.267419 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":23,"thread_start_us":86,"threads_started":1}
I20260928 12:21:44.267547 9276 tablet_service.cc:1467] Processing CreateTablet for tablet 19c822da2082490d94637fdaae448d20 (DEFAULT_TABLE table=test-table [id=82e5a42e465d4304b4ce03d45e0a3035]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:44.267916 9276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19c822da2082490d94637fdaae448d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:44.268227 9413 tablet_service.cc:1467] Processing CreateTablet for tablet 19c822da2082490d94637fdaae448d20 (DEFAULT_TABLE table=test-table [id=82e5a42e465d4304b4ce03d45e0a3035]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:44.268551 9413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19c822da2082490d94637fdaae448d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:44.270730 9641 tablet_bootstrap.cc:492] T 19c822da2082490d94637fdaae448d20 P 53c647c83fb847d090c0cc0c1e3e6f60: Bootstrap starting.
I20260928 12:21:44.271337 9642 tablet_bootstrap.cc:492] T 19c822da2082490d94637fdaae448d20 P cf83a5232738461e9eace3fc21dd0929: Bootstrap starting.
I20260928 12:21:44.271485 9641 tablet_bootstrap.cc:654] T 19c822da2082490d94637fdaae448d20 P 53c647c83fb847d090c0cc0c1e3e6f60: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:44.271759 9641 log.cc:824] T 19c822da2082490d94637fdaae448d20 P 53c647c83fb847d090c0cc0c1e3e6f60: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:44.271953 9642 tablet_bootstrap.cc:654] T 19c822da2082490d94637fdaae448d20 P cf83a5232738461e9eace3fc21dd0929: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:44.272140 9644 tablet_bootstrap.cc:492] T 19c822da2082490d94637fdaae448d20 P 9c6a797c45334b15bca72df411c5d1bb: Bootstrap starting.
I20260928 12:21:44.272276 9642 log.cc:824] T 19c822da2082490d94637fdaae448d20 P cf83a5232738461e9eace3fc21dd0929: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:44.272678 9644 tablet_bootstrap.cc:654] T 19c822da2082490d94637fdaae448d20 P 9c6a797c45334b15bca72df411c5d1bb: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:44.272944 9644 log.cc:824] T 19c822da2082490d94637fdaae448d20 P 9c6a797c45334b15bca72df411c5d1bb: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:44.272961 9642 tablet_bootstrap.cc:492] T 19c822da2082490d94637fdaae448d20 P cf83a5232738461e9eace3fc21dd0929: No bootstrap required, opened a new log
I20260928 12:21:44.273010 9642 ts_tablet_manager.cc:1397] T 19c822da2082490d94637fdaae448d20 P cf83a5232738461e9eace3fc21dd0929: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:21:44.273633 9641 tablet_bootstrap.cc:492] T 19c822da2082490d94637fdaae448d20 P 53c647c83fb847d090c0cc0c1e3e6f60: No bootstrap required, opened a new log
I20260928 12:21:44.273633 9644 tablet_bootstrap.cc:492] T 19c822da2082490d94637fdaae448d20 P 9c6a797c45334b15bca72df411c5d1bb: No bootstrap required, opened a new log
I20260928 12:21:44.273700 9644 ts_tablet_manager.cc:1397] T 19c822da2082490d94637fdaae448d20 P 9c6a797c45334b15bca72df411c5d1bb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:21:44.273700 9641 ts_tablet_manager.cc:1397] T 19c822da2082490d94637fdaae448d20 P 53c647c83fb847d090c0cc0c1e3e6f60: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 12:21:44.274960 9642 raft_consensus.cc:348] T 19c822da2082490d94637fdaae448d20 P cf83a5232738461e9eace3fc21dd0929 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c647c83fb847d090c0cc0c1e3e6f60" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42075 } } peers { permanent_uuid: "cf83a5232738461e9eace3fc21dd0929" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 43087 } } peers { permanent_uuid: "9c6a797c45334b15bca72df411c5d1bb" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34785 } }
I20260928 12:21:44.275090 9642 raft_consensus.cc:374] T 19c822da2082490d94637fdaae448d20 P cf83a5232738461e9eace3fc21dd0929 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:44.275120 9642 raft_consensus.cc:729] T 19c822da2082490d94637fdaae448d20 P cf83a5232738461e9eace3fc21dd0929 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf83a5232738461e9eace3fc21dd0929, State: Initialized, Role: FOLLOWER
I20260928 12:21:44.275238 9642 consensus_queue.cc:260] T 19c822da2082490d94637fdaae448d20 P cf83a5232738461e9eace3fc21dd0929 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c647c83fb847d090c0cc0c1e3e6f60" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42075 } } peers { permanent_uuid: "cf83a5232738461e9eace3fc21dd0929" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 43087 } } peers { permanent_uuid: "9c6a797c45334b15bca72df411c5d1bb" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34785 } }
I20260928 12:21:44.275480 9642 ts_tablet_manager.cc:1428] T 19c822da2082490d94637fdaae448d20 P cf83a5232738461e9eace3fc21dd0929: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 12:21:44.275629 9616 heartbeater.cc:503] Master 127.2.56.62:43477 was elected leader, sending a full tablet report...
I20260928 12:21:44.275645 9641 raft_consensus.cc:348] T 19c822da2082490d94637fdaae448d20 P 53c647c83fb847d090c0cc0c1e3e6f60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c647c83fb847d090c0cc0c1e3e6f60" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42075 } } peers { permanent_uuid: "cf83a5232738461e9eace3fc21dd0929" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 43087 } } peers { permanent_uuid: "9c6a797c45334b15bca72df411c5d1bb" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34785 } }
I20260928 12:21:44.275796 9641 raft_consensus.cc:374] T 19c822da2082490d94637fdaae448d20 P 53c647c83fb847d090c0cc0c1e3e6f60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:44.275827 9641 raft_consensus.cc:729] T 19c822da2082490d94637fdaae448d20 P 53c647c83fb847d090c0cc0c1e3e6f60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c647c83fb847d090c0cc0c1e3e6f60, State: Initialized, Role: FOLLOWER
I20260928 12:21:44.275848 9644 raft_consensus.cc:348] T 19c822da2082490d94637fdaae448d20 P 9c6a797c45334b15bca72df411c5d1bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c647c83fb847d090c0cc0c1e3e6f60" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42075 } } peers { permanent_uuid: "cf83a5232738461e9eace3fc21dd0929" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 43087 } } peers { permanent_uuid: "9c6a797c45334b15bca72df411c5d1bb" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34785 } }
I20260928 12:21:44.275967 9644 raft_consensus.cc:374] T 19c822da2082490d94637fdaae448d20 P 9c6a797c45334b15bca72df411c5d1bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:44.275997 9644 raft_consensus.cc:729] T 19c822da2082490d94637fdaae448d20 P 9c6a797c45334b15bca72df411c5d1bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c6a797c45334b15bca72df411c5d1bb, State: Initialized, Role: FOLLOWER
I20260928 12:21:44.275969 9641 consensus_queue.cc:260] T 19c822da2082490d94637fdaae448d20 P 53c647c83fb847d090c0cc0c1e3e6f60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c647c83fb847d090c0cc0c1e3e6f60" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42075 } } peers { permanent_uuid: "cf83a5232738461e9eace3fc21dd0929" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 43087 } } peers { permanent_uuid: "9c6a797c45334b15bca72df411c5d1bb" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34785 } }
I20260928 12:21:44.276117 9644 consensus_queue.cc:260] T 19c822da2082490d94637fdaae448d20 P 9c6a797c45334b15bca72df411c5d1bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c647c83fb847d090c0cc0c1e3e6f60" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42075 } } peers { permanent_uuid: "cf83a5232738461e9eace3fc21dd0929" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 43087 } } peers { permanent_uuid: "9c6a797c45334b15bca72df411c5d1bb" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34785 } }
I20260928 12:21:44.276216 9641 ts_tablet_manager.cc:1428] T 19c822da2082490d94637fdaae448d20 P 53c647c83fb847d090c0cc0c1e3e6f60: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 12:21:44.276343 9343 heartbeater.cc:503] Master 127.2.56.62:43477 was elected leader, sending a full tablet report...
I20260928 12:21:44.276342 9644 ts_tablet_manager.cc:1428] T 19c822da2082490d94637fdaae448d20 P 9c6a797c45334b15bca72df411c5d1bb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:21:44.276458 9480 heartbeater.cc:503] Master 127.2.56.62:43477 was elected leader, sending a full tablet report...
W20260928 12:21:44.353137 9482 tablet.cc:2369] T 19c822da2082490d94637fdaae448d20 P 9c6a797c45334b15bca72df411c5d1bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:21:44.379050 9647 raft_consensus.cc:482] T 19c822da2082490d94637fdaae448d20 P cf83a5232738461e9eace3fc21dd0929 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:21:44.379222 9647 raft_consensus.cc:504] T 19c822da2082490d94637fdaae448d20 P cf83a5232738461e9eace3fc21dd0929 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c647c83fb847d090c0cc0c1e3e6f60" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42075 } } peers { permanent_uuid: "cf83a5232738461e9eace3fc21dd0929" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 43087 } } peers { permanent_uuid: "9c6a797c45334b15bca72df411c5d1bb" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34785 } }
I20260928 12:21:44.379632 9647 leader_election.cc:288] T 19c822da2082490d94637fdaae448d20 P cf83a5232738461e9eace3fc21dd0929 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53c647c83fb847d090c0cc0c1e3e6f60 (127.2.56.1:42075), 9c6a797c45334b15bca72df411c5d1bb (127.2.56.2:34785)
I20260928 12:21:44.382722 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:44.379994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:55033 (local address 127.2.56.2:34785)
0928 12:21:44.380050 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:44.380052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:44.380060 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:21:44.380085 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:44.380086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:44.380095 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:44.380140 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:44.380272 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.380753 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.381799 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.382370 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.382437 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:44.382451 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:44.382606 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 12:21:44.382650 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 12:21:44.382735 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:44.379869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:51981 (local address 127.2.56.1:42075)
0928 12:21:44.379935 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:44.379937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:44.379949 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:21:44.379982 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:44.379983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:44.379997 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:44.380053 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:44.380220 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.380722 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.381779 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.382365 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.382441 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:44.382451 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:44.382633 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 12:21:44.382676 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 12:21:44.382730 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:44.379889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:34785 (local address 127.2.56.3:55033)
0928 12:21:44.379931 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:44.379940 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:44.379974 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:44.380153 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:44.380153 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:44.380155 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:44.380216 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:44.380217 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.380822 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.380823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:44.381706 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:44.381709 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.382437 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.382438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:44.382535 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:44.382539 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:21:44.382592 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 12:21:44.382633 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":1}
I20260928 12:21:44.382740 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:44.379793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:42075 (local address 127.2.56.3:51981)
0928 12:21:44.379849 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:44.379863 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:44.379923 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:44.380067 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:44.380068 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:44.380070 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:44.380162 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:44.380164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.380796 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.380798 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:44.381692 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:44.381695 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.382437 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.382438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:44.382535 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:44.382539 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:21:44.382592 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 12:21:44.382633 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":40}
I20260928 12:21:44.383147 9433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c822da2082490d94637fdaae448d20" candidate_uuid: "cf83a5232738461e9eace3fc21dd0929" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c6a797c45334b15bca72df411c5d1bb" is_pre_election: true
I20260928 12:21:44.383363 9433 raft_consensus.cc:2454] T 19c822da2082490d94637fdaae448d20 P 9c6a797c45334b15bca72df411c5d1bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf83a5232738461e9eace3fc21dd0929 in term 0.
I20260928 12:21:44.383428 9296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c822da2082490d94637fdaae448d20" candidate_uuid: "cf83a5232738461e9eace3fc21dd0929" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53c647c83fb847d090c0cc0c1e3e6f60" is_pre_election: true
I20260928 12:21:44.383591 9296 raft_consensus.cc:2454] T 19c822da2082490d94637fdaae448d20 P 53c647c83fb847d090c0cc0c1e3e6f60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf83a5232738461e9eace3fc21dd0929 in term 0.
I20260928 12:21:44.383671 9505 leader_election.cc:302] T 19c822da2082490d94637fdaae448d20 P cf83a5232738461e9eace3fc21dd0929 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c6a797c45334b15bca72df411c5d1bb, cf83a5232738461e9eace3fc21dd0929; no voters:
I20260928 12:21:44.383853 9647 raft_consensus.cc:2781] T 19c822da2082490d94637fdaae448d20 P cf83a5232738461e9eace3fc21dd0929 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:21:44.383908 9647 raft_consensus.cc:482] T 19c822da2082490d94637fdaae448d20 P cf83a5232738461e9eace3fc21dd0929 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:21:44.383939 9647 raft_consensus.cc:3037] T 19c822da2082490d94637fdaae448d20 P cf83a5232738461e9eace3fc21dd0929 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:44.384742 9647 raft_consensus.cc:504] T 19c822da2082490d94637fdaae448d20 P cf83a5232738461e9eace3fc21dd0929 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c647c83fb847d090c0cc0c1e3e6f60" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42075 } } peers { permanent_uuid: "cf83a5232738461e9eace3fc21dd0929" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 43087 } } peers { permanent_uuid: "9c6a797c45334b15bca72df411c5d1bb" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34785 } }
I20260928 12:21:44.384869 9647 leader_election.cc:288] T 19c822da2082490d94637fdaae448d20 P cf83a5232738461e9eace3fc21dd0929 [CANDIDATE]: Term 1 election: Requested vote from peers 53c647c83fb847d090c0cc0c1e3e6f60 (127.2.56.1:42075), 9c6a797c45334b15bca72df411c5d1bb (127.2.56.2:34785)
I20260928 12:21:44.385040 9296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c822da2082490d94637fdaae448d20" candidate_uuid: "cf83a5232738461e9eace3fc21dd0929" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53c647c83fb847d090c0cc0c1e3e6f60"
I20260928 12:21:44.385073 9433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c822da2082490d94637fdaae448d20" candidate_uuid: "cf83a5232738461e9eace3fc21dd0929" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c6a797c45334b15bca72df411c5d1bb"
I20260928 12:21:44.385126 9296 raft_consensus.cc:3037] T 19c822da2082490d94637fdaae448d20 P 53c647c83fb847d090c0cc0c1e3e6f60 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:44.385161 9433 raft_consensus.cc:3037] T 19c822da2082490d94637fdaae448d20 P 9c6a797c45334b15bca72df411c5d1bb [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:44.386114 9433 raft_consensus.cc:2454] T 19c822da2082490d94637fdaae448d20 P 9c6a797c45334b15bca72df411c5d1bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf83a5232738461e9eace3fc21dd0929 in term 1.
I20260928 12:21:44.386114 9296 raft_consensus.cc:2454] T 19c822da2082490d94637fdaae448d20 P 53c647c83fb847d090c0cc0c1e3e6f60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf83a5232738461e9eace3fc21dd0929 in term 1.
I20260928 12:21:44.386281 9505 leader_election.cc:302] T 19c822da2082490d94637fdaae448d20 P cf83a5232738461e9eace3fc21dd0929 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c6a797c45334b15bca72df411c5d1bb, cf83a5232738461e9eace3fc21dd0929; no voters:
I20260928 12:21:44.386395 9647 raft_consensus.cc:2781] T 19c822da2082490d94637fdaae448d20 P cf83a5232738461e9eace3fc21dd0929 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:21:44.386571 9647 raft_consensus.cc:686] T 19c822da2082490d94637fdaae448d20 P cf83a5232738461e9eace3fc21dd0929 [term 1 LEADER]: Becoming Leader. State: Replica: cf83a5232738461e9eace3fc21dd0929, State: Running, Role: LEADER
I20260928 12:21:44.386724 9647 consensus_queue.cc:237] T 19c822da2082490d94637fdaae448d20 P cf83a5232738461e9eace3fc21dd0929 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c647c83fb847d090c0cc0c1e3e6f60" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42075 } } peers { permanent_uuid: "cf83a5232738461e9eace3fc21dd0929" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 43087 } } peers { permanent_uuid: "9c6a797c45334b15bca72df411c5d1bb" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34785 } }
I20260928 12:21:44.388150 8379 catalog_manager.cc:5495] T 19c822da2082490d94637fdaae448d20 P cf83a5232738461e9eace3fc21dd0929 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf83a5232738461e9eace3fc21dd0929 (127.2.56.3). New cstate: current_term: 1 leader_uuid: "cf83a5232738461e9eace3fc21dd0929" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c647c83fb847d090c0cc0c1e3e6f60" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf83a5232738461e9eace3fc21dd0929" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 43087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c6a797c45334b15bca72df411c5d1bb" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34785 } health_report { overall_health: UNKNOWN } } }
I20260928 12:21:44.411501 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:44.409149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54132 (local address 127.2.56.3:43087)
0928 12:21:44.409208 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:44.409210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:44.409218 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:21:44.409256 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:44.409257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:44.409302 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:44.409364 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:21:44.409493 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.410006 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.410754 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.410969 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.411023 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:44.411165 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:21:44.411288 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:21:44.411364 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:44.411380 (+ 16us) server_negotiation.cc:299] Negotiation successful
0928 12:21:44.411403 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 12:21:44.413527 9433 raft_consensus.cc:1261] T 19c822da2082490d94637fdaae448d20 P 9c6a797c45334b15bca72df411c5d1bb [term 1 FOLLOWER]: Refusing update from remote peer cf83a5232738461e9eace3fc21dd0929: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:21:44.413564 9296 raft_consensus.cc:1261] T 19c822da2082490d94637fdaae448d20 P 53c647c83fb847d090c0cc0c1e3e6f60 [term 1 FOLLOWER]: Refusing update from remote peer cf83a5232738461e9eace3fc21dd0929: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:21:44.413892 9650 consensus_queue.cc:1035] T 19c822da2082490d94637fdaae448d20 P cf83a5232738461e9eace3fc21dd0929 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c6a797c45334b15bca72df411c5d1bb" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:21:44.414197 9650 consensus_queue.cc:1035] T 19c822da2082490d94637fdaae448d20 P cf83a5232738461e9eace3fc21dd0929 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53c647c83fb847d090c0cc0c1e3e6f60" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:21:44.416030 9348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:44.413190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44178 (local address 127.2.56.60:44031)
0928 12:21:44.413238 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:44.413240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:44.413248 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:21:44.413296 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:44.413297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:44.413324 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:44.413384 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:44.413569 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.414172 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.415148 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.415724 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.415799 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:44.415885 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:44.415912 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 12:21:44.415960 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:21:44.416036 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:44.413039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:44031 (local address 127.0.0.1:44178)
0928 12:21:44.413090 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:44.413102 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:44.413149 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:44.413439 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:44.413440 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:44.413443 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:44.413496 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:44.413498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.414243 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.414244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:44.415065 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:44.415067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.415792 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.415793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:44.415864 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:44.415868 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:21:44.415909 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 12:21:44.415962 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 12:21:44.417400 9170 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.10] Dedup: 1.7->[1.8-1.10]
I20260928 12:21:44.418299 9659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 262a43ba51f54d5ba1fc806531019e58. Latest consensus state: current_term: 1 leader_uuid: "262a43ba51f54d5ba1fc806531019e58" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: false } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: NON_VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: true } } }
I20260928 12:21:44.418395 9659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [sys.catalog]: This master's current role is: LEARNER
I20260928 12:21:44.419507 9657 mvcc.cc:205] Tried to move back new op lower bound from 7334289835674746880 to 7334289835568967680. Current Snapshot: MvccSnapshot[applied={T|T < 7334289835674746880}]
I20260928 12:21:44.419723 9655 mvcc.cc:205] Tried to move back new op lower bound from 7334289835674746880 to 7334289835568967680. Current Snapshot: MvccSnapshot[applied={T|T < 7334289835674746880}]
I20260928 12:21:44.422446 8380 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:21:44.422531 8380 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:21:44.423183 9652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "262a43ba51f54d5ba1fc806531019e58" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: false } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: NON_VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: true } } }
I20260928 12:21:44.423278 9652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:44.423631 9127 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58: attempting to promote NON_VOTER 35a79f2366a84381b917b76c334e55bc to VOTER
I20260928 12:21:44.423774 9127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: false } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: false } }
I20260928 12:21:44.424188 9170 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [term 1 LEARNER]: Refusing update from remote peer 262a43ba51f54d5ba1fc806531019e58: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20260928 12:21:44.424288 8987 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [term 1 FOLLOWER]: Refusing update from remote peer 262a43ba51f54d5ba1fc806531019e58: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20260928 12:21:44.424454 8380 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58: Sending DeleteTablet for 3 replicas of tablet 19c822da2082490d94637fdaae448d20
I20260928 12:21:44.424476 8939 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20260928 12:21:44.424593 8939 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260928 12:21:44.424829 9413 tablet_service.cc:1514] Processing DeleteTablet for tablet 19c822da2082490d94637fdaae448d20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:44 UTC) from {username='oryx', principal='oryx/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:51708
I20260928 12:21:44.424865 9549 tablet_service.cc:1514] Processing DeleteTablet for tablet 19c822da2082490d94637fdaae448d20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:44 UTC) from {username='oryx', principal='oryx/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:54116
I20260928 12:21:44.425083 9670 tablet_replica.cc:330] T 19c822da2082490d94637fdaae448d20 P cf83a5232738461e9eace3fc21dd0929: stopping tablet replica
I20260928 12:21:44.425146 9276 tablet_service.cc:1514] Processing DeleteTablet for tablet 19c822da2082490d94637fdaae448d20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:44 UTC) from {username='oryx', principal='oryx/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:38852
I20260928 12:21:44.425190 9670 raft_consensus.cc:2229] T 19c822da2082490d94637fdaae448d20 P cf83a5232738461e9eace3fc21dd0929 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:21:44.425339 9670 raft_consensus.cc:2258] T 19c822da2082490d94637fdaae448d20 P cf83a5232738461e9eace3fc21dd0929 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:44.425220 8939 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 7 to 12, 35a79f2366a84381b917b76c334e55bc (127.2.56.60) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: false } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: false } } }
I20260928 12:21:44.425866 9670 ts_tablet_manager.cc:1905] T 19c822da2082490d94637fdaae448d20 P cf83a5232738461e9eace3fc21dd0929: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:44.426261 8987 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 7 to 12, 35a79f2366a84381b917b76c334e55bc (127.2.56.60) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: false } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: false } } }
I20260928 12:21:44.426359 9671 tablet_replica.cc:330] T 19c822da2082490d94637fdaae448d20 P 9c6a797c45334b15bca72df411c5d1bb: stopping tablet replica
I20260928 12:21:44.426450 9671 raft_consensus.cc:2229] T 19c822da2082490d94637fdaae448d20 P 9c6a797c45334b15bca72df411c5d1bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:21:44.426445 8939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "262a43ba51f54d5ba1fc806531019e58" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: false } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: false } } }
I20260928 12:21:44.426522 8939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:44.426560 9671 raft_consensus.cc:2258] T 19c822da2082490d94637fdaae448d20 P 9c6a797c45334b15bca72df411c5d1bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:44.426556 8939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "262a43ba51f54d5ba1fc806531019e58" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: false } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: false } } }
I20260928 12:21:44.426591 8939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:44.426617 8939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "262a43ba51f54d5ba1fc806531019e58" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: false } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: false } } }
I20260928 12:21:44.426652 8939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 262a43ba51f54d5ba1fc806531019e58 [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:44.426703 9672 tablet_replica.cc:330] T 19c822da2082490d94637fdaae448d20 P 53c647c83fb847d090c0cc0c1e3e6f60: stopping tablet replica
I20260928 12:21:44.426779 9672 raft_consensus.cc:2229] T 19c822da2082490d94637fdaae448d20 P 53c647c83fb847d090c0cc0c1e3e6f60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:21:44.426857 9672 raft_consensus.cc:2258] T 19c822da2082490d94637fdaae448d20 P 53c647c83fb847d090c0cc0c1e3e6f60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:44.426971 9170 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 7 to 12, 35a79f2366a84381b917b76c334e55bc (127.2.56.60) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: false } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: false } } }
I20260928 12:21:44.427668 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 8348
I20260928 12:21:44.427716 9670 ts_tablet_manager.cc:1918] T 19c822da2082490d94637fdaae448d20 P cf83a5232738461e9eace3fc21dd0929: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:21:44.427776 9670 log.cc:1196] T 19c822da2082490d94637fdaae448d20 P cf83a5232738461e9eace3fc21dd0929: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-2/wal/wals/19c822da2082490d94637fdaae448d20
I20260928 12:21:44.428050 9670 ts_tablet_manager.cc:1939] T 19c822da2082490d94637fdaae448d20 P cf83a5232738461e9eace3fc21dd0929: Deleting consensus metadata
I20260928 12:21:44.428833 9659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "262a43ba51f54d5ba1fc806531019e58" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: false } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: false } } }
I20260928 12:21:44.428908 9659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:21:44.429226 9672 ts_tablet_manager.cc:1905] T 19c822da2082490d94637fdaae448d20 P 53c647c83fb847d090c0cc0c1e3e6f60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:44.429390 9674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "262a43ba51f54d5ba1fc806531019e58" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: false } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: false } } }
I20260928 12:21:44.429483 9674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:21:44.430621 9672 ts_tablet_manager.cc:1918] T 19c822da2082490d94637fdaae448d20 P 53c647c83fb847d090c0cc0c1e3e6f60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:21:44.430684 9672 log.cc:1196] T 19c822da2082490d94637fdaae448d20 P 53c647c83fb847d090c0cc0c1e3e6f60: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-0/wal/wals/19c822da2082490d94637fdaae448d20
I20260928 12:21:44.430886 9672 ts_tablet_manager.cc:1939] T 19c822da2082490d94637fdaae448d20 P 53c647c83fb847d090c0cc0c1e3e6f60: Deleting consensus metadata
I20260928 12:21:44.432000 9671 ts_tablet_manager.cc:1905] T 19c822da2082490d94637fdaae448d20 P 9c6a797c45334b15bca72df411c5d1bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:44.438356 9348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:44.425818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44190 (local address 127.2.56.60:44031)
0928 12:21:44.425857 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:44.425859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:44.427321 (+ 1462us) server_negotiation.cc:407] Connection header received
0928 12:21:44.427345 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:44.427346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:44.427357 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:44.427405 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:44.431769 (+ 4364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.432473 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.433562 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.433827 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.433894 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:44.435493 (+ 1599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:44.435495 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:44.435495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:44.435504 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:44.437046 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:44.437484 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:44.437485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:44.437486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:44.437544 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:44.437729 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:44.437729 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:44.437730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:44.437858 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:44.437953 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:44.438064 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 12:21:44.438177 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 12:21:44.438351 9671 ts_tablet_manager.cc:1918] T 19c822da2082490d94637fdaae448d20 P 9c6a797c45334b15bca72df411c5d1bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:21:44.438441 9671 log.cc:1196] T 19c822da2082490d94637fdaae448d20 P 9c6a797c45334b15bca72df411c5d1bb: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-1/wal/wals/19c822da2082490d94637fdaae448d20
I20260928 12:21:44.438724 9671 ts_tablet_manager.cc:1939] T 19c822da2082490d94637fdaae448d20 P 9c6a797c45334b15bca72df411c5d1bb: Deleting consensus metadata
W20260928 12:21:44.439391 9150 connection.cc:535] server connection from 127.0.0.1:44178 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 12:21:44.439419 9367 connection.cc:535] server connection from 127.0.0.1:51708 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 12:21:44.439553 9503 connection.cc:535] server connection from 127.0.0.1:54116 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 12:21:44.439805 8963 connection.cc:535] server connection from 127.0.0.1:35308 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 12:21:44.440063 9231 connection.cc:535] server connection from 127.0.0.1:38852 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 12:21:44.448282 9348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:44.441314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44194 (local address 127.2.56.60:44031)
0928 12:21:44.441349 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:44.441351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:44.441442 (+ 91us) server_negotiation.cc:407] Connection header received
0928 12:21:44.441498 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:44.441499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:44.441509 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:44.441561 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:44.442034 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.442811 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.443520 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.443825 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.443886 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:44.445736 (+ 1850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:44.445738 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:44.445738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:44.445745 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:44.447259 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:44.447550 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:44.447552 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:44.447552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:44.447606 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:44.447760 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:44.447761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:44.447761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:44.447866 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:44.447960 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:44.448093 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 12:21:44.448178 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:21:44.452018 9028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:44.441212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35330 (local address 127.2.56.61:42657)
0928 12:21:44.441368 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:44.441370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:44.445489 (+ 4119us) server_negotiation.cc:407] Connection header received
0928 12:21:44.445595 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:44.445596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:44.445611 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:44.445674 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:44.446249 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.447010 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.448024 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:44.448303 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:44.448364 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:44.449570 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:44.449571 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:44.449571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:44.449578 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:44.451003 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:44.451325 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:44.451325 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:44.451326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:44.451383 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:44.451545 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:44.451545 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:44.451545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:44.451661 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:44.451750 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:44.451841 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 12:21:44.451919 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 12:21:45.421025 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:45.420627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.2.56.3:38451)
0928 12:21:45.420831 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:45.420952 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":82,"threads_started":1}
I20260928 12:21:45.421181 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:45.420699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.2.56.2:48341)
0928 12:21:45.421025 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:45.421112 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":50,"threads_started":1}
I20260928 12:21:45.421463 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:45.420875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.2.56.1:56453)
0928 12:21:45.421307 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:45.421408 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":64,"threads_started":1}
W20260928 12:21:45.421494 9480 heartbeater.cc:644] Failed to heartbeat to 127.2.56.62:43477 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
W20260928 12:21:45.421689 9343 heartbeater.cc:644] Failed to heartbeat to 127.2.56.62:43477 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
W20260928 12:21:45.421815 9616 heartbeater.cc:644] Failed to heartbeat to 127.2.56.62:43477 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
I20260928 12:21:45.927404 9684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 262a43ba51f54d5ba1fc806531019e58)
I20260928 12:21:45.927515 9685 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 262a43ba51f54d5ba1fc806531019e58)
I20260928 12:21:45.927592 9684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: false } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: false } }
I20260928 12:21:45.927659 9685 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: false } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: false } }
I20260928 12:21:45.927942 9684 leader_election.cc:288] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 262a43ba51f54d5ba1fc806531019e58 (127.2.56.62:43477), 35a79f2366a84381b917b76c334e55bc (127.2.56.60:44031)
I20260928 12:21:45.928170 9685 leader_election.cc:288] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 262a43ba51f54d5ba1fc806531019e58 (127.2.56.62:43477), 2b6ccb471f0e419c91dd33e886b9fe85 (127.2.56.61:42657)
I20260928 12:21:45.928387 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:45.928075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.0.0.1:58806)
0928 12:21:45.928270 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:45.928335 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":53,"threads_started":1}
I20260928 12:21:45.928493 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:45.928075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.0.0.1:58808)
0928 12:21:45.928194 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:45.928255 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":53,"threads_started":1}
W20260928 12:21:45.928565 8963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
W20260928 12:21:45.928603 9149 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
I20260928 12:21:45.929126 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:45.929041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.0.0.1:58836)
0928 12:21:45.929075 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:45.929096 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 12:21:45.929215 9149 leader_election.cc:334] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 262a43ba51f54d5ba1fc806531019e58 (127.2.56.62:43477): Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
I20260928 12:21:45.929158 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:45.928997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.0.0.1:58828)
0928 12:21:45.929105 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:45.929129 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":71,"threads_started":1}
W20260928 12:21:45.929301 8963 leader_election.cc:334] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 262a43ba51f54d5ba1fc806531019e58 (127.2.56.62:43477): Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
I20260928 12:21:45.931288 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:45.928159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:44031 (local address 127.0.0.1:57434)
0928 12:21:45.928486 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:45.928495 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:45.928562 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:45.928759 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:45.928760 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:45.928762 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:45.928826 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:45.928829 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:45.929689 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:45.929690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:45.930506 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:45.930508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:45.931061 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:45.931062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:45.931116 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:45.931120 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:21:45.931165 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 12:21:45.931213 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":44,"threads_started":1}
I20260928 12:21:45.931325 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:45.928359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57434 (local address 127.2.56.60:44031)
0928 12:21:45.928441 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:45.928443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:45.928610 (+ 167us) server_negotiation.cc:407] Connection header received
0928 12:21:45.928634 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:45.928635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:45.928645 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:45.928711 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:45.928891 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:45.929617 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:45.930590 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:45.930998 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:45.931055 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:45.931069 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:45.931174 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 12:21:45.931213 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66,"thread_start_us":42,"threads_started":1}
I20260928 12:21:45.931348 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:45.928089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.61:42657 (local address 127.0.0.1:55152)
0928 12:21:45.928288 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:45.928298 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:45.928365 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:45.928759 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:45.928760 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:45.928762 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:45.928825 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:45.928828 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:45.929684 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:45.929686 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:45.930478 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:45.930481 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:45.931106 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:45.931107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:45.931162 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:45.931165 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:21:45.931208 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 12:21:45.931260 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":45,"threads_started":1}
I20260928 12:21:45.931365 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:45.928221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55152 (local address 127.2.56.61:42657)
0928 12:21:45.928571 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:45.928573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:45.928581 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:21:45.928611 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:45.928612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:45.928623 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:45.928699 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:45.928900 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:45.929613 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:45.930557 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:45.931034 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:45.931085 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:45.931099 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:45.931220 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 12:21:45.931259 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":66,"threads_started":1}
I20260928 12:21:45.931761 8987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "35a79f2366a84381b917b76c334e55bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" is_pre_election: true
I20260928 12:21:45.931813 9170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "35a79f2366a84381b917b76c334e55bc" is_pre_election: true
I20260928 12:21:45.931903 9170 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b6ccb471f0e419c91dd33e886b9fe85 in term 1.
I20260928 12:21:45.931887 8987 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35a79f2366a84381b917b76c334e55bc in term 1.
I20260928 12:21:45.932053 9147 leader_election.cc:302] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2b6ccb471f0e419c91dd33e886b9fe85, 35a79f2366a84381b917b76c334e55bc; no voters: 262a43ba51f54d5ba1fc806531019e58
I20260928 12:21:45.932053 8962 leader_election.cc:302] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2b6ccb471f0e419c91dd33e886b9fe85, 35a79f2366a84381b917b76c334e55bc; no voters: 262a43ba51f54d5ba1fc806531019e58
I20260928 12:21:45.932168 9685 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 12:21:45.932176 9684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 12:21:45.932202 9685 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [term 1 FOLLOWER]: Starting leader election (detected failure of leader 262a43ba51f54d5ba1fc806531019e58)
I20260928 12:21:45.932220 9685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [term 1 FOLLOWER]: Advancing to term 2
I20260928 12:21:45.932214 9684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 262a43ba51f54d5ba1fc806531019e58)
I20260928 12:21:45.932235 9684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [term 1 FOLLOWER]: Advancing to term 2
I20260928 12:21:45.932993 9684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: false } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: false } }
I20260928 12:21:45.932996 9685 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [term 2 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: false } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: false } }
I20260928 12:21:45.933111 9684 leader_election.cc:288] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [CANDIDATE]: Term 2 election: Requested vote from peers 262a43ba51f54d5ba1fc806531019e58 (127.2.56.62:43477), 35a79f2366a84381b917b76c334e55bc (127.2.56.60:44031)
I20260928 12:21:45.933117 9685 leader_election.cc:288] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [CANDIDATE]: Term 2 election: Requested vote from peers 262a43ba51f54d5ba1fc806531019e58 (127.2.56.62:43477), 2b6ccb471f0e419c91dd33e886b9fe85 (127.2.56.61:42657)
I20260928 12:21:45.933239 8987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "35a79f2366a84381b917b76c334e55bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2b6ccb471f0e419c91dd33e886b9fe85"
I20260928 12:21:45.933338 8987 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 35a79f2366a84381b917b76c334e55bc in current term 2: Already voted for candidate 2b6ccb471f0e419c91dd33e886b9fe85 in this term.
I20260928 12:21:45.933382 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:45.933198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.0.0.1:58844)
0928 12:21:45.933312 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:45.933339 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260928 12:21:45.933444 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:45.933216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.0.0.1:58850)
0928 12:21:45.933397 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:45.933422 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260928 12:21:45.933377 9170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "35a79f2366a84381b917b76c334e55bc"
I20260928 12:21:45.933498 9170 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b6ccb471f0e419c91dd33e886b9fe85 in current term 2: Already voted for candidate 35a79f2366a84381b917b76c334e55bc in this term.
I20260928 12:21:45.933758 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:45.933618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.0.0.1:58858)
0928 12:21:45.933716 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:45.933737 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 12:21:45.933780 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:45.933707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.0.0.1:58860)
0928 12:21:45.933739 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:45.933759 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260928 12:21:45.933806 9149 leader_election.cc:334] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 262a43ba51f54d5ba1fc806531019e58 (127.2.56.62:43477): Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
W20260928 12:21:45.933820 8963 leader_election.cc:334] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 262a43ba51f54d5ba1fc806531019e58 (127.2.56.62:43477): Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
I20260928 12:21:45.933831 9149 leader_election.cc:302] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 35a79f2366a84381b917b76c334e55bc; no voters: 262a43ba51f54d5ba1fc806531019e58, 2b6ccb471f0e419c91dd33e886b9fe85
I20260928 12:21:45.933847 8963 leader_election.cc:302] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b6ccb471f0e419c91dd33e886b9fe85; no voters: 262a43ba51f54d5ba1fc806531019e58, 35a79f2366a84381b917b76c334e55bc
I20260928 12:21:45.933944 9685 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 12:21:45.934016 9684 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 12:21:46.422379 9695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:46.421860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.2.56.2:41097)
0928 12:21:46.422253 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:46.422307 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":118,"threads_started":1}
I20260928 12:21:46.422432 9696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:46.422021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.2.56.1:58173)
0928 12:21:46.422316 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:46.422359 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":73,"threads_started":1}
I20260928 12:21:46.422605 9697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:46.422084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.2.56.3:43861)
0928 12:21:46.422482 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:46.422556 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":59,"threads_started":1}
I20260928 12:21:47.423280 9699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:47.422971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.2.56.2:41669)
0928 12:21:47.423178 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:47.423215 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":88,"threads_started":1}
I20260928 12:21:47.423270 9698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:47.422984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.2.56.1:34277)
0928 12:21:47.423157 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:47.423202 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":75,"threads_started":1}
I20260928 12:21:47.423270 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:47.423058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.2.56.3:44129)
0928 12:21:47.423157 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:47.423202 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":50,"threads_started":1}
I20260928 12:21:47.779414 9701 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:21:47.779536 9701 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: false } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: false } }
I20260928 12:21:47.779711 9701 leader_election.cc:288] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 262a43ba51f54d5ba1fc806531019e58 (127.2.56.62:43477), 35a79f2366a84381b917b76c334e55bc (127.2.56.60:44031)
I20260928 12:21:47.779964 9170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "35a79f2366a84381b917b76c334e55bc" is_pre_election: true
I20260928 12:21:47.780061 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:47.779847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.0.0.1:59118)
0928 12:21:47.779958 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:47.780007 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":66,"threads_started":1}
I20260928 12:21:47.780098 9170 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b6ccb471f0e419c91dd33e886b9fe85 in term 2.
I20260928 12:21:47.780284 8962 leader_election.cc:302] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b6ccb471f0e419c91dd33e886b9fe85, 35a79f2366a84381b917b76c334e55bc; no voters:
I20260928 12:21:47.780398 9701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260928 12:21:47.780455 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:47.780361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.0.0.1:59130)
0928 12:21:47.780402 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:47.780428 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 12:21:47.780460 9701 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:21:47.780506 9701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [term 2 FOLLOWER]: Advancing to term 3
W20260928 12:21:47.780512 8963 leader_election.cc:334] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 262a43ba51f54d5ba1fc806531019e58 (127.2.56.62:43477): Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
I20260928 12:21:47.781467 9701 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: false } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: false } }
I20260928 12:21:47.781598 9701 leader_election.cc:288] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [CANDIDATE]: Term 3 election: Requested vote from peers 262a43ba51f54d5ba1fc806531019e58 (127.2.56.62:43477), 35a79f2366a84381b917b76c334e55bc (127.2.56.60:44031)
I20260928 12:21:47.781802 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:47.781698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.0.0.1:59134)
0928 12:21:47.781736 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:47.781769 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260928 12:21:47.781798 9170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "35a79f2366a84381b917b76c334e55bc"
I20260928 12:21:47.781883 9170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [term 2 FOLLOWER]: Advancing to term 3
I20260928 12:21:47.782155 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:47.782065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.0.0.1:59144)
0928 12:21:47.782106 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:47.782130 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 12:21:47.782233 8963 leader_election.cc:334] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 262a43ba51f54d5ba1fc806531019e58 (127.2.56.62:43477): Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
I20260928 12:21:47.782668 9170 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b6ccb471f0e419c91dd33e886b9fe85 in term 3.
I20260928 12:21:47.782856 8962 leader_election.cc:302] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2b6ccb471f0e419c91dd33e886b9fe85, 35a79f2366a84381b917b76c334e55bc; no voters: 262a43ba51f54d5ba1fc806531019e58
I20260928 12:21:47.782954 9701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 12:21:47.783056 9701 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [term 3 LEADER]: Becoming Leader. State: Replica: 2b6ccb471f0e419c91dd33e886b9fe85, State: Running, Role: LEADER
I20260928 12:21:47.783126 9701 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: false } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: false } }
I20260928 12:21:47.783528 9704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b6ccb471f0e419c91dd33e886b9fe85. Latest consensus state: current_term: 3 leader_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: false } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: false } } }
I20260928 12:21:47.783635 9704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:47.783834 9706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:21:47.784435 9706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:21:47.784519 9706 catalog_manager.cc:1259] Loaded cluster ID: f5f2d04c9c0b48a1b4ff09425dfdc0bd
I20260928 12:21:47.784555 9706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:21:47.784765 9706 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:21:47.784848 9706 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85: Loaded TSK: 0
I20260928 12:21:47.785002 9706 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:21:47.863722 8971 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:21:47.865665 8971 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:21:47.866739 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:47.866621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.0.0.1:59168)
0928 12:21:47.866670 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:47.866703 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 12:21:47.866761 9170 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [term 3 FOLLOWER]: Refusing update from remote peer 2b6ccb471f0e419c91dd33e886b9fe85: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260928 12:21:47.866999 9704 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 12:21:47.867146 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:47.867022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.0.0.1:59184)
0928 12:21:47.867065 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:47.867124 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 12:21:47.867254 8963 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 -> Peer 262a43ba51f54d5ba1fc806531019e58 (127.2.56.62:43477): Couldn't send request to peer 262a43ba51f54d5ba1fc806531019e58. Status: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 12:21:47.867713 9708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b6ccb471f0e419c91dd33e886b9fe85. Latest consensus state: current_term: 3 leader_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: false } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: false } } }
I20260928 12:21:47.867812 9708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:21:47.868104 9707 mvcc.cc:205] Tried to move back new op lower bound from 7334289849820217344 to 7334289849480368128. Current Snapshot: MvccSnapshot[applied={T|T < 7334289849820217344}]
I20260928 12:21:47.868201 9701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: false } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: false } } }
I20260928 12:21:47.868288 9701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85 [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:47.868353 9709 mvcc.cc:205] Tried to move back new op lower bound from 7334289849820217344 to 7334289849480368128. Current Snapshot: MvccSnapshot[applied={T|T < 7334289849820217344}]
I20260928 12:21:47.868438 9708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "262a43ba51f54d5ba1fc806531019e58" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43477 } } peers { permanent_uuid: "2b6ccb471f0e419c91dd33e886b9fe85" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 42657 } attrs { promote: false } } peers { permanent_uuid: "35a79f2366a84381b917b76c334e55bc" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 44031 } attrs { promote: false } } }
I20260928 12:21:47.868510 9708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35a79f2366a84381b917b76c334e55bc [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:21:47.873675 9718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:47.870452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57368 (local address 127.2.56.3:43087)
0928 12:21:47.870568 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:47.870571 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:47.870690 (+ 119us) server_negotiation.cc:407] Connection header received
0928 12:21:47.870722 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:47.870723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:47.870735 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:47.870959 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:47.871002 (+ 43us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:47.871542 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:47.872811 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:47.873359 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:47.873427 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:47.873442 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:47.873560 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 12:21:47.873601 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":57,"threads_started":1}
I20260928 12:21:47.873672 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:47.870221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:43087 (local address 127.0.0.1:57368)
0928 12:21:47.870548 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:47.870558 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:47.870619 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:47.870832 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:47.870833 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:47.870834 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:47.870891 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:47.870892 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:47.871611 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:47.871612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:47.872713 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:47.872715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:47.873435 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:47.873436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:47.873501 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:47.873507 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 12:21:47.873549 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 12:21:47.873601 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":3,"thread_start_us":67,"threads_started":1}
I20260928 12:21:47.873932 9549 tablet_service.cc:1467] Processing CreateTablet for tablet ae423a906ede4fecb65199a2b3e8375f (DEFAULT_TABLE table=test-table [id=d083e4b705f74d6bbaae0ad7bdeaf10f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:47.874044 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:47.870299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:42075 (local address 127.0.0.1:38760)
0928 12:21:47.870692 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:47.870700 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:47.870741 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:47.870964 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:47.870964 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:47.870967 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:47.871000 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:47.871002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:47.871872 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:47.871873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:47.872991 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:47.872993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:47.873835 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:47.873835 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:47.873894 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:47.873897 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:21:47.873930 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 12:21:47.873975 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":33,"thread_start_us":72,"threads_started":1}
I20260928 12:21:47.874074 9549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae423a906ede4fecb65199a2b3e8375f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:47.874087 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:47.870284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48946 (local address 127.2.56.2:34785)
0928 12:21:47.870413 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:47.870416 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:47.870558 (+ 142us) server_negotiation.cc:407] Connection header received
0928 12:21:47.870642 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:47.870643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:47.870656 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:47.870728 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:47.870987 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:47.871774 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:47.872814 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:47.873367 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:47.873416 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:47.873432 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:47.873619 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 12:21:47.873659 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":74,"threads_started":1}
I20260928 12:21:47.874294 9714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:47.870214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:34785 (local address 127.0.0.1:48946)
0928 12:21:47.870487 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:47.870497 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:47.870559 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:47.870784 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:47.870785 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:47.870787 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:47.870850 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:47.870852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:47.871867 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:47.871868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:47.872722 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:47.872724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:47.873485 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:47.873486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:47.873546 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:47.873550 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:21:47.874186 (+ 636us) client_negotiation.cc:240] Negotiation successful
0928 12:21:47.874231 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":37}
I20260928 12:21:47.874321 9717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:47.870372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38760 (local address 127.2.56.1:42075)
0928 12:21:47.870499 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:47.870500 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:47.870798 (+ 298us) server_negotiation.cc:407] Connection header received
0928 12:21:47.870834 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:47.870835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:47.870847 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:47.870915 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:47.871060 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:47.871802 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:47.873065 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:47.873676 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:47.873733 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:47.874184 (+ 451us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:47.874203 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 12:21:47.874250 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":81,"threads_started":1}
I20260928 12:21:47.874504 9413 tablet_service.cc:1467] Processing CreateTablet for tablet ae423a906ede4fecb65199a2b3e8375f (DEFAULT_TABLE table=test-table [id=d083e4b705f74d6bbaae0ad7bdeaf10f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:47.874629 9413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae423a906ede4fecb65199a2b3e8375f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:47.875487 9719 tablet_bootstrap.cc:492] T ae423a906ede4fecb65199a2b3e8375f P cf83a5232738461e9eace3fc21dd0929: Bootstrap starting.
I20260928 12:21:47.875912 9719 tablet_bootstrap.cc:654] T ae423a906ede4fecb65199a2b3e8375f P cf83a5232738461e9eace3fc21dd0929: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:47.876071 9720 tablet_bootstrap.cc:492] T ae423a906ede4fecb65199a2b3e8375f P 9c6a797c45334b15bca72df411c5d1bb: Bootstrap starting.
I20260928 12:21:47.876128 9276 tablet_service.cc:1467] Processing CreateTablet for tablet ae423a906ede4fecb65199a2b3e8375f (DEFAULT_TABLE table=test-table [id=d083e4b705f74d6bbaae0ad7bdeaf10f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:47.876248 9276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae423a906ede4fecb65199a2b3e8375f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:47.876431 9720 tablet_bootstrap.cc:654] T ae423a906ede4fecb65199a2b3e8375f P 9c6a797c45334b15bca72df411c5d1bb: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:47.877022 9720 tablet_bootstrap.cc:492] T ae423a906ede4fecb65199a2b3e8375f P 9c6a797c45334b15bca72df411c5d1bb: No bootstrap required, opened a new log
I20260928 12:21:47.877072 9720 ts_tablet_manager.cc:1397] T ae423a906ede4fecb65199a2b3e8375f P 9c6a797c45334b15bca72df411c5d1bb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:21:47.877238 9720 raft_consensus.cc:348] T ae423a906ede4fecb65199a2b3e8375f P 9c6a797c45334b15bca72df411c5d1bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a797c45334b15bca72df411c5d1bb" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34785 } } peers { permanent_uuid: "cf83a5232738461e9eace3fc21dd0929" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 43087 } } peers { permanent_uuid: "53c647c83fb847d090c0cc0c1e3e6f60" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42075 } }
I20260928 12:21:47.877347 9720 raft_consensus.cc:374] T ae423a906ede4fecb65199a2b3e8375f P 9c6a797c45334b15bca72df411c5d1bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:47.877369 9720 raft_consensus.cc:729] T ae423a906ede4fecb65199a2b3e8375f P 9c6a797c45334b15bca72df411c5d1bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c6a797c45334b15bca72df411c5d1bb, State: Initialized, Role: FOLLOWER
I20260928 12:21:47.877430 9720 consensus_queue.cc:260] T ae423a906ede4fecb65199a2b3e8375f P 9c6a797c45334b15bca72df411c5d1bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a797c45334b15bca72df411c5d1bb" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34785 } } peers { permanent_uuid: "cf83a5232738461e9eace3fc21dd0929" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 43087 } } peers { permanent_uuid: "53c647c83fb847d090c0cc0c1e3e6f60" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42075 } }
I20260928 12:21:47.877470 9719 tablet_bootstrap.cc:492] T ae423a906ede4fecb65199a2b3e8375f P cf83a5232738461e9eace3fc21dd0929: No bootstrap required, opened a new log
I20260928 12:21:47.877521 9719 ts_tablet_manager.cc:1397] T ae423a906ede4fecb65199a2b3e8375f P cf83a5232738461e9eace3fc21dd0929: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:21:47.877646 9720 ts_tablet_manager.cc:1428] T ae423a906ede4fecb65199a2b3e8375f P 9c6a797c45334b15bca72df411c5d1bb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:21:47.877664 9719 raft_consensus.cc:348] T ae423a906ede4fecb65199a2b3e8375f P cf83a5232738461e9eace3fc21dd0929 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a797c45334b15bca72df411c5d1bb" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34785 } } peers { permanent_uuid: "cf83a5232738461e9eace3fc21dd0929" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 43087 } } peers { permanent_uuid: "53c647c83fb847d090c0cc0c1e3e6f60" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42075 } }
I20260928 12:21:47.877739 9719 raft_consensus.cc:374] T ae423a906ede4fecb65199a2b3e8375f P cf83a5232738461e9eace3fc21dd0929 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:47.877758 9719 raft_consensus.cc:729] T ae423a906ede4fecb65199a2b3e8375f P cf83a5232738461e9eace3fc21dd0929 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf83a5232738461e9eace3fc21dd0929, State: Initialized, Role: FOLLOWER
I20260928 12:21:47.877807 9719 consensus_queue.cc:260] T ae423a906ede4fecb65199a2b3e8375f P cf83a5232738461e9eace3fc21dd0929 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a797c45334b15bca72df411c5d1bb" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34785 } } peers { permanent_uuid: "cf83a5232738461e9eace3fc21dd0929" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 43087 } } peers { permanent_uuid: "53c647c83fb847d090c0cc0c1e3e6f60" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42075 } }
I20260928 12:21:47.877980 9719 ts_tablet_manager.cc:1428] T ae423a906ede4fecb65199a2b3e8375f P cf83a5232738461e9eace3fc21dd0929: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:47.877983 9699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:47.877868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.2.56.2:46639)
0928 12:21:47.877924 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:47.877954 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 12:21:47.878105 9723 tablet_bootstrap.cc:492] T ae423a906ede4fecb65199a2b3e8375f P 53c647c83fb847d090c0cc0c1e3e6f60: Bootstrap starting.
I20260928 12:21:47.878252 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:47.878161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.2.56.3:44047)
0928 12:21:47.878199 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:47.878227 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 12:21:47.878595 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:47.878474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.2.56.3:56219)
0928 12:21:47.878546 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:47.878571 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 12:21:47.878888 9617 heartbeater.cc:503] Master 127.2.56.61:42657 was elected leader, sending a full tablet report...
I20260928 12:21:47.879163 9723 tablet_bootstrap.cc:654] T ae423a906ede4fecb65199a2b3e8375f P 53c647c83fb847d090c0cc0c1e3e6f60: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:47.879519 9481 heartbeater.cc:503] Master 127.2.56.61:42657 was elected leader, sending a full tablet report...
I20260928 12:21:47.879710 9699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:47.879624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.2.56.2:40557)
0928 12:21:47.879658 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:47.879687 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 12:21:47.880056 9723 tablet_bootstrap.cc:492] T ae423a906ede4fecb65199a2b3e8375f P 53c647c83fb847d090c0cc0c1e3e6f60: No bootstrap required, opened a new log
I20260928 12:21:47.880097 9723 ts_tablet_manager.cc:1397] T ae423a906ede4fecb65199a2b3e8375f P 53c647c83fb847d090c0cc0c1e3e6f60: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:21:47.880232 9723 raft_consensus.cc:348] T ae423a906ede4fecb65199a2b3e8375f P 53c647c83fb847d090c0cc0c1e3e6f60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a797c45334b15bca72df411c5d1bb" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34785 } } peers { permanent_uuid: "cf83a5232738461e9eace3fc21dd0929" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 43087 } } peers { permanent_uuid: "53c647c83fb847d090c0cc0c1e3e6f60" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42075 } }
I20260928 12:21:47.880287 9723 raft_consensus.cc:374] T ae423a906ede4fecb65199a2b3e8375f P 53c647c83fb847d090c0cc0c1e3e6f60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:47.880307 9723 raft_consensus.cc:729] T ae423a906ede4fecb65199a2b3e8375f P 53c647c83fb847d090c0cc0c1e3e6f60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c647c83fb847d090c0cc0c1e3e6f60, State: Initialized, Role: FOLLOWER
I20260928 12:21:47.880357 9723 consensus_queue.cc:260] T ae423a906ede4fecb65199a2b3e8375f P 53c647c83fb847d090c0cc0c1e3e6f60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a797c45334b15bca72df411c5d1bb" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34785 } } peers { permanent_uuid: "cf83a5232738461e9eace3fc21dd0929" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 43087 } } peers { permanent_uuid: "53c647c83fb847d090c0cc0c1e3e6f60" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42075 } }
I20260928 12:21:47.880530 9723 ts_tablet_manager.cc:1428] T ae423a906ede4fecb65199a2b3e8375f P 53c647c83fb847d090c0cc0c1e3e6f60: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:47.880761 9698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:47.880671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.2.56.1:50193)
0928 12:21:47.880709 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:47.880733 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 12:21:47.880973 9344 heartbeater.cc:503] Master 127.2.56.61:42657 was elected leader, sending a full tablet report...
I20260928 12:21:47.881021 9698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:47.880949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.2.56.1:53323)
0928 12:21:47.880974 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:47.880997 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260928 12:21:47.951565 9345 tablet.cc:2369] T ae423a906ede4fecb65199a2b3e8375f P 53c647c83fb847d090c0cc0c1e3e6f60: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:21:47.991031 9618 tablet.cc:2369] T ae423a906ede4fecb65199a2b3e8375f P cf83a5232738461e9eace3fc21dd0929: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:21:48.098896 9727 raft_consensus.cc:482] T ae423a906ede4fecb65199a2b3e8375f P 53c647c83fb847d090c0cc0c1e3e6f60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:21:48.099017 9727 raft_consensus.cc:504] T ae423a906ede4fecb65199a2b3e8375f P 53c647c83fb847d090c0cc0c1e3e6f60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a797c45334b15bca72df411c5d1bb" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34785 } } peers { permanent_uuid: "cf83a5232738461e9eace3fc21dd0929" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 43087 } } peers { permanent_uuid: "53c647c83fb847d090c0cc0c1e3e6f60" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42075 } }
I20260928 12:21:48.099436 9727 leader_election.cc:288] T ae423a906ede4fecb65199a2b3e8375f P 53c647c83fb847d090c0cc0c1e3e6f60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c6a797c45334b15bca72df411c5d1bb (127.2.56.2:34785), cf83a5232738461e9eace3fc21dd0929 (127.2.56.3:43087)
I20260928 12:21:48.103129 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:48.099784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:48283 (local address 127.2.56.2:34785)
0928 12:21:48.099831 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:48.099833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:48.099907 (+ 74us) server_negotiation.cc:407] Connection header received
0928 12:21:48.099981 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:48.099982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:48.099994 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:48.100029 (+ 35us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:48.100128 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:48.100881 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:48.102440 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:48.102826 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:48.102886 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:48.102897 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:48.103023 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 12:21:48.103068 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 12:21:48.103132 9728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:48.099612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:34785 (local address 127.2.56.1:48283)
0928 12:21:48.099853 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:48.099861 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:48.099923 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:48.100041 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:48.100042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:48.100044 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:48.100081 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:48.100082 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:48.100961 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:48.100962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:48.102343 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:48.102347 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:48.102891 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:48.102891 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:48.102953 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:48.102956 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:21:48.103000 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 12:21:48.103048 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":110,"thread_start_us":138,"threads_started":1}
I20260928 12:21:48.103387 9433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae423a906ede4fecb65199a2b3e8375f" candidate_uuid: "53c647c83fb847d090c0cc0c1e3e6f60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c6a797c45334b15bca72df411c5d1bb" is_pre_election: true
I20260928 12:21:48.103494 9433 raft_consensus.cc:2454] T ae423a906ede4fecb65199a2b3e8375f P 9c6a797c45334b15bca72df411c5d1bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53c647c83fb847d090c0cc0c1e3e6f60 in term 0.
I20260928 12:21:48.103700 9232 leader_election.cc:302] T ae423a906ede4fecb65199a2b3e8375f P 53c647c83fb847d090c0cc0c1e3e6f60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53c647c83fb847d090c0cc0c1e3e6f60, 9c6a797c45334b15bca72df411c5d1bb; no voters:
I20260928 12:21:48.103817 9698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:48.099609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:43087 (local address 127.2.56.1:59157)
0928 12:21:48.099657 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:48.099670 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:48.099737 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:48.099921 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:48.099922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:48.099925 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:48.100002 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:48.100004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:48.100966 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:48.100967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:48.102343 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:48.102347 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:48.103507 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:48.103508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:48.103574 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:48.103579 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:21:48.103627 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 12:21:48.103664 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":30}
I20260928 12:21:48.103842 9718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:48.099674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:59157 (local address 127.2.56.3:43087)
0928 12:21:48.099729 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:48.099731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:48.099740 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:21:48.099824 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:48.099825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:48.099839 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:48.099902 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:48.100064 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:48.100894 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:48.102948 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:48.103411 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:48.103471 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:48.103483 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:48.103735 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 12:21:48.103781 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 12:21:48.103878 9727 raft_consensus.cc:2781] T ae423a906ede4fecb65199a2b3e8375f P 53c647c83fb847d090c0cc0c1e3e6f60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:21:48.103907 9727 raft_consensus.cc:482] T ae423a906ede4fecb65199a2b3e8375f P 53c647c83fb847d090c0cc0c1e3e6f60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:21:48.103935 9727 raft_consensus.cc:3037] T ae423a906ede4fecb65199a2b3e8375f P 53c647c83fb847d090c0cc0c1e3e6f60 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:48.104090 9569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae423a906ede4fecb65199a2b3e8375f" candidate_uuid: "53c647c83fb847d090c0cc0c1e3e6f60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf83a5232738461e9eace3fc21dd0929" is_pre_election: true
I20260928 12:21:48.104228 9569 raft_consensus.cc:2454] T ae423a906ede4fecb65199a2b3e8375f P cf83a5232738461e9eace3fc21dd0929 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53c647c83fb847d090c0cc0c1e3e6f60 in term 0.
I20260928 12:21:48.104816 9727 raft_consensus.cc:504] T ae423a906ede4fecb65199a2b3e8375f P 53c647c83fb847d090c0cc0c1e3e6f60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a797c45334b15bca72df411c5d1bb" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34785 } } peers { permanent_uuid: "cf83a5232738461e9eace3fc21dd0929" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 43087 } } peers { permanent_uuid: "53c647c83fb847d090c0cc0c1e3e6f60" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42075 } }
I20260928 12:21:48.104949 9727 leader_election.cc:288] T ae423a906ede4fecb65199a2b3e8375f P 53c647c83fb847d090c0cc0c1e3e6f60 [CANDIDATE]: Term 1 election: Requested vote from peers 9c6a797c45334b15bca72df411c5d1bb (127.2.56.2:34785), cf83a5232738461e9eace3fc21dd0929 (127.2.56.3:43087)
I20260928 12:21:48.105085 9433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae423a906ede4fecb65199a2b3e8375f" candidate_uuid: "53c647c83fb847d090c0cc0c1e3e6f60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c6a797c45334b15bca72df411c5d1bb"
I20260928 12:21:48.105158 9433 raft_consensus.cc:3037] T ae423a906ede4fecb65199a2b3e8375f P 9c6a797c45334b15bca72df411c5d1bb [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:48.105855 9433 raft_consensus.cc:2454] T ae423a906ede4fecb65199a2b3e8375f P 9c6a797c45334b15bca72df411c5d1bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53c647c83fb847d090c0cc0c1e3e6f60 in term 1.
I20260928 12:21:48.105963 9569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae423a906ede4fecb65199a2b3e8375f" candidate_uuid: "53c647c83fb847d090c0cc0c1e3e6f60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf83a5232738461e9eace3fc21dd0929"
I20260928 12:21:48.105998 9232 leader_election.cc:302] T ae423a906ede4fecb65199a2b3e8375f P 53c647c83fb847d090c0cc0c1e3e6f60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53c647c83fb847d090c0cc0c1e3e6f60, 9c6a797c45334b15bca72df411c5d1bb; no voters:
I20260928 12:21:48.106038 9569 raft_consensus.cc:3037] T ae423a906ede4fecb65199a2b3e8375f P cf83a5232738461e9eace3fc21dd0929 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:48.106086 9727 raft_consensus.cc:2781] T ae423a906ede4fecb65199a2b3e8375f P 53c647c83fb847d090c0cc0c1e3e6f60 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:21:48.106251 9727 raft_consensus.cc:686] T ae423a906ede4fecb65199a2b3e8375f P 53c647c83fb847d090c0cc0c1e3e6f60 [term 1 LEADER]: Becoming Leader. State: Replica: 53c647c83fb847d090c0cc0c1e3e6f60, State: Running, Role: LEADER
I20260928 12:21:48.106323 9727 consensus_queue.cc:237] T ae423a906ede4fecb65199a2b3e8375f P 53c647c83fb847d090c0cc0c1e3e6f60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a797c45334b15bca72df411c5d1bb" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34785 } } peers { permanent_uuid: "cf83a5232738461e9eace3fc21dd0929" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 43087 } } peers { permanent_uuid: "53c647c83fb847d090c0cc0c1e3e6f60" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42075 } }
I20260928 12:21:48.106943 9569 raft_consensus.cc:2454] T ae423a906ede4fecb65199a2b3e8375f P cf83a5232738461e9eace3fc21dd0929 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53c647c83fb847d090c0cc0c1e3e6f60 in term 1.
I20260928 12:21:48.107259 9698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:48.107169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.2.56.1:56585)
0928 12:21:48.107205 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:48.107236 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260928 12:21:48.107168 8977 catalog_manager.cc:5495] T ae423a906ede4fecb65199a2b3e8375f P 53c647c83fb847d090c0cc0c1e3e6f60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53c647c83fb847d090c0cc0c1e3e6f60 (127.2.56.1). New cstate: current_term: 1 leader_uuid: "53c647c83fb847d090c0cc0c1e3e6f60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a797c45334b15bca72df411c5d1bb" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf83a5232738461e9eace3fc21dd0929" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 43087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53c647c83fb847d090c0cc0c1e3e6f60" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42075 } health_report { overall_health: HEALTHY } } }
I20260928 12:21:48.155942 9717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:48.153773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38770 (local address 127.2.56.1:42075)
0928 12:21:48.153830 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:48.153832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:48.153864 (+ 32us) server_negotiation.cc:407] Connection header received
0928 12:21:48.153913 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:48.153914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:48.153939 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:48.153999 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:21:48.154124 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:48.154623 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:48.155332 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:48.155520 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:48.155565 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:48.155664 (+ 99us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:21:48.155780 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:21:48.155831 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:48.155846 (+ 15us) server_negotiation.cc:299] Negotiation successful
0928 12:21:48.155865 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 12:21:48.157121 9433 raft_consensus.cc:1261] T ae423a906ede4fecb65199a2b3e8375f P 9c6a797c45334b15bca72df411c5d1bb [term 1 FOLLOWER]: Refusing update from remote peer 53c647c83fb847d090c0cc0c1e3e6f60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:21:48.157244 9569 raft_consensus.cc:1261] T ae423a906ede4fecb65199a2b3e8375f P cf83a5232738461e9eace3fc21dd0929 [term 1 FOLLOWER]: Refusing update from remote peer 53c647c83fb847d090c0cc0c1e3e6f60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:21:48.157590 9727 consensus_queue.cc:1035] T ae423a906ede4fecb65199a2b3e8375f P 53c647c83fb847d090c0cc0c1e3e6f60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf83a5232738461e9eace3fc21dd0929" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 43087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:21:48.157743 9729 consensus_queue.cc:1035] T ae423a906ede4fecb65199a2b3e8375f P 53c647c83fb847d090c0cc0c1e3e6f60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c6a797c45334b15bca72df411c5d1bb" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:21:48.158497 9699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:48.158293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.2.56.2:59943)
0928 12:21:48.158418 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:48.158461 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260928 12:21:48.158815 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:48.158709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.2.56.3:48809)
0928 12:21:48.158757 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:48.158782 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 12:21:48.159686 9698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:48.159585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.2.56.1:36881)
0928 12:21:48.159630 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:48.159653 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 12:21:48.161446 9699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:48.159921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.2.56.2:50875)
0928 12:21:48.161369 (+ 1448us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:48.161404 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1432}
I20260928 12:21:48.161525 8977 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:21:48.161617 8977 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:21:48.162639 9698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:48.162469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.2.56.1:35577)
0928 12:21:48.162581 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:48.162608 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260928 12:21:48.163479 8977 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b6ccb471f0e419c91dd33e886b9fe85: Sending DeleteTablet for 3 replicas of tablet ae423a906ede4fecb65199a2b3e8375f
I20260928 12:21:48.163687 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:48.163402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43477 (local address 127.2.56.3:50083)
0928 12:21:48.163629 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:48.163661 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:43477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210}
I20260928 12:21:48.163854 9276 tablet_service.cc:1514] Processing DeleteTablet for tablet ae423a906ede4fecb65199a2b3e8375f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:48 UTC) from {username='oryx', principal='oryx/127.2.56.61@KRBTEST.COM'} at 127.0.0.1:38760
I20260928 12:21:48.163873 9735 mvcc.cc:205] Tried to move back new op lower bound from 7334289851009507328 to 7334289850804281344. Current Snapshot: MvccSnapshot[applied={T|T < 7334289851009507328 or (T in {7334289851009507328})}]
I20260928 12:21:48.163995 9413 tablet_service.cc:1514] Processing DeleteTablet for tablet ae423a906ede4fecb65199a2b3e8375f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:48 UTC) from {username='oryx', principal='oryx/127.2.56.61@KRBTEST.COM'} at 127.0.0.1:48946
I20260928 12:21:48.164152 9743 tablet_replica.cc:330] T ae423a906ede4fecb65199a2b3e8375f P 53c647c83fb847d090c0cc0c1e3e6f60: stopping tablet replica
I20260928 12:21:48.164203 9744 tablet_replica.cc:330] T ae423a906ede4fecb65199a2b3e8375f P 9c6a797c45334b15bca72df411c5d1bb: stopping tablet replica
I20260928 12:21:48.164211 9743 raft_consensus.cc:2229] T ae423a906ede4fecb65199a2b3e8375f P 53c647c83fb847d090c0cc0c1e3e6f60 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:21:48.164247 9744 raft_consensus.cc:2229] T ae423a906ede4fecb65199a2b3e8375f P 9c6a797c45334b15bca72df411c5d1bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:21:48.164283 9744 raft_consensus.cc:2258] T ae423a906ede4fecb65199a2b3e8375f P 9c6a797c45334b15bca72df411c5d1bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:48.164283 9743 raft_consensus.cc:2258] T ae423a906ede4fecb65199a2b3e8375f P 53c647c83fb847d090c0cc0c1e3e6f60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:48.164296 9549 tablet_service.cc:1514] Processing DeleteTablet for tablet ae423a906ede4fecb65199a2b3e8375f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:48 UTC) from {username='oryx', principal='oryx/127.2.56.61@KRBTEST.COM'} at 127.0.0.1:57368
I20260928 12:21:48.164419 9548 tablet_service.cc:1514] Processing DeleteTablet for tablet ae423a906ede4fecb65199a2b3e8375f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:48 UTC) from {username='oryx', principal='oryx/127.2.56.61@KRBTEST.COM'} at 127.0.0.1:57368
I20260928 12:21:48.164559 9743 ts_tablet_manager.cc:1905] T ae423a906ede4fecb65199a2b3e8375f P 53c647c83fb847d090c0cc0c1e3e6f60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:48.164919 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 9124
I20260928 12:21:48.165699 9744 ts_tablet_manager.cc:1905] T ae423a906ede4fecb65199a2b3e8375f P 9c6a797c45334b15bca72df411c5d1bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:48.167402 9744 ts_tablet_manager.cc:1918] T ae423a906ede4fecb65199a2b3e8375f P 9c6a797c45334b15bca72df411c5d1bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:21:48.167454 9744 log.cc:1196] T ae423a906ede4fecb65199a2b3e8375f P 9c6a797c45334b15bca72df411c5d1bb: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-1/wal/wals/ae423a906ede4fecb65199a2b3e8375f
I20260928 12:21:48.167657 9744 ts_tablet_manager.cc:1939] T ae423a906ede4fecb65199a2b3e8375f P 9c6a797c45334b15bca72df411c5d1bb: Deleting consensus metadata
I20260928 12:21:48.167845 9745 tablet_replica.cc:330] T ae423a906ede4fecb65199a2b3e8375f P cf83a5232738461e9eace3fc21dd0929: stopping tablet replica
I20260928 12:21:48.167894 9745 raft_consensus.cc:2229] T ae423a906ede4fecb65199a2b3e8375f P cf83a5232738461e9eace3fc21dd0929 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:21:48.167946 9745 raft_consensus.cc:2258] T ae423a906ede4fecb65199a2b3e8375f P cf83a5232738461e9eace3fc21dd0929 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:48.168290 8964 catalog_manager.cc:4841] TS 9c6a797c45334b15bca72df411c5d1bb (127.2.56.2:34785): tablet ae423a906ede4fecb65199a2b3e8375f (table test-table [id=d083e4b705f74d6bbaae0ad7bdeaf10f]) successfully deleted
I20260928 12:21:48.168340 9745 ts_tablet_manager.cc:1905] T ae423a906ede4fecb65199a2b3e8375f P cf83a5232738461e9eace3fc21dd0929: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:48.169898 9745 ts_tablet_manager.cc:1918] T ae423a906ede4fecb65199a2b3e8375f P cf83a5232738461e9eace3fc21dd0929: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:21:48.169956 9745 log.cc:1196] T ae423a906ede4fecb65199a2b3e8375f P cf83a5232738461e9eace3fc21dd0929: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598072638673-2272-0/minicluster-data/ts-2/wal/wals/ae423a906ede4fecb65199a2b3e8375f
I20260928 12:21:48.170151 9745 ts_tablet_manager.cc:1939] T ae423a906ede4fecb65199a2b3e8375f P cf83a5232738461e9eace3fc21dd0929: Deleting consensus metadata
I20260928 12:21:48.170450 8962 catalog_manager.cc:4841] TS cf83a5232738461e9eace3fc21dd0929 (127.2.56.3:43087): tablet ae423a906ede4fecb65199a2b3e8375f (table test-table [id=d083e4b705f74d6bbaae0ad7bdeaf10f]) successfully deleted
W20260928 12:21:48.170593 8962 catalog_manager.cc:4808] TS cf83a5232738461e9eace3fc21dd0929 (127.2.56.3:43087): delete failed for tablet ae423a906ede4fecb65199a2b3e8375f because the tablet was not found. No further retry: Not found: Tablet not found: ae423a906ede4fecb65199a2b3e8375f
I20260928 12:21:48.172365 9714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:48.172242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:42075 (local address 127.0.0.1:38776)
0928 12:21:48.172297 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:48.172330 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.1:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 12:21:48.172441 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 9350
W20260928 12:21:48.172519 8964 catalog_manager.cc:4584] TS 53c647c83fb847d090c0cc0c1e3e6f60 (127.2.56.1:42075): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ae423a906ede4fecb65199a2b3e8375f: Network error: Client connection negotiation failed: client connection to 127.2.56.1:42075: connect: Connection refused (error 111)
I20260928 12:21:48.177335 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 9486
I20260928 12:21:48.182775 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 8844
I20260928 12:21:48.187186 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 9031
2026-09-28T12:21:48Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6709 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:21:48 dist-test-slave-920p krb5kdc[9748](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:21:48 dist-test-slave-920p krb5kdc[9748](info): set up 2 sockets
Sep 28 12:21:48 dist-test-slave-920p krb5kdc[9748](info): commencing operation
krb5kdc: starting...
W20260928 12:21:49.145951 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:21:49 dist-test-slave-920p krb5kdc[9748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:21:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:21:49Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.62.keytab.
Entry for principal oryx/127.2.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.62.keytab.
I20260928 12:21:49.268390 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:42227
--webserver_interface=127.2.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.62.keytab
--principal=oryx/127.2.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:41561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:42227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:49.353868 9761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:49.354105 9761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:49.354125 9761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:49.355266 9761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:21:49.355298 9761 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 12:21:49.355309 9761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:49.355320 9761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:49.355334 9761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:21:49.355345 9761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:21:49.356825 9761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:41561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:42227
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.62.keytab
--principal=oryx/127.2.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:42227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:49.357043 9761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:49.357235 9761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:49.362582 9767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:49.362756 9761 server_base.cc:1034] running on GCE node
W20260928 12:21:49.362639 9766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:49.362644 9769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:49.363199 9761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:49.363509 9761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:49.364655 9761 hybrid_clock.cc:648] HybridClock initialized: now 1790598109364638 us; error 34 us; skew 500 ppm
Sep 28 12:21:49 dist-test-slave-920p krb5kdc[9748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598109, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:49.367185 9761 init.cc:377] Logged in from keytab as oryx/127.2.56.62@KRBTEST.COM (short username oryx)
I20260928 12:21:49.367620 9761 webserver.cc:460] Webserver started at http://127.2.56.62:37545/ using document root <none> and password file <none>
I20260928 12:21:49.367841 9761 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:49.367887 9761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:49.368017 9761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:49.369146 9761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "049b783e402e4560b7d3b9b60de4b55b"
format_stamp: "Formatted at 2026-09-28 12:21:49 on dist-test-slave-920p"
server_key: "97768c0982f7b4bbd688f485c7160ce8"
server_key_iv: "8e12b3901bd356cf995dd78631e05c84"
server_key_version: "encryptionkey@0"
I20260928 12:21:49.369532 9761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "049b783e402e4560b7d3b9b60de4b55b"
format_stamp: "Formatted at 2026-09-28 12:21:49 on dist-test-slave-920p"
server_key: "97768c0982f7b4bbd688f485c7160ce8"
server_key_iv: "8e12b3901bd356cf995dd78631e05c84"
server_key_version: "encryptionkey@0"
I20260928 12:21:49.370819 9761 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:21:49.371447 9776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:49.371609 9761 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:49.371693 9761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "049b783e402e4560b7d3b9b60de4b55b"
format_stamp: "Formatted at 2026-09-28 12:21:49 on dist-test-slave-920p"
server_key: "97768c0982f7b4bbd688f485c7160ce8"
server_key_iv: "8e12b3901bd356cf995dd78631e05c84"
server_key_version: "encryptionkey@0"
I20260928 12:21:49.371758 9761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:49.385406 9761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:49.387549 9761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:49.387686 9761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:49.392539 9761 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:42227
I20260928 12:21:49.392616 9838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:42227 every 8 connection(s)
I20260928 12:21:49.392951 9761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:21:49.393680 9839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:49.396126 9839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 049b783e402e4560b7d3b9b60de4b55b: Bootstrap starting.
I20260928 12:21:49.396730 9839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 049b783e402e4560b7d3b9b60de4b55b: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:49.396996 9839 log.cc:824] T 00000000000000000000000000000000 P 049b783e402e4560b7d3b9b60de4b55b: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:49.397655 9839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 049b783e402e4560b7d3b9b60de4b55b: No bootstrap required, opened a new log
I20260928 12:21:49.399765 9839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 049b783e402e4560b7d3b9b60de4b55b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "049b783e402e4560b7d3b9b60de4b55b" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 42227 } }
I20260928 12:21:49.399931 9839 raft_consensus.cc:374] T 00000000000000000000000000000000 P 049b783e402e4560b7d3b9b60de4b55b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:49.399960 9839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 049b783e402e4560b7d3b9b60de4b55b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 049b783e402e4560b7d3b9b60de4b55b, State: Initialized, Role: FOLLOWER
I20260928 12:21:49.400091 9839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 049b783e402e4560b7d3b9b60de4b55b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "049b783e402e4560b7d3b9b60de4b55b" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 42227 } }
I20260928 12:21:49.400169 9839 raft_consensus.cc:388] T 00000000000000000000000000000000 P 049b783e402e4560b7d3b9b60de4b55b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:21:49.400199 9839 raft_consensus.cc:482] T 00000000000000000000000000000000 P 049b783e402e4560b7d3b9b60de4b55b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:21:49.400233 9839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 049b783e402e4560b7d3b9b60de4b55b [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:49.400733 9839 raft_consensus.cc:504] T 00000000000000000000000000000000 P 049b783e402e4560b7d3b9b60de4b55b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "049b783e402e4560b7d3b9b60de4b55b" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 42227 } }
I20260928 12:21:49.400823 9839 leader_election.cc:302] T 00000000000000000000000000000000 P 049b783e402e4560b7d3b9b60de4b55b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 049b783e402e4560b7d3b9b60de4b55b; no voters:
I20260928 12:21:49.400970 9839 leader_election.cc:288] T 00000000000000000000000000000000 P 049b783e402e4560b7d3b9b60de4b55b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:21:49.401018 9842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 049b783e402e4560b7d3b9b60de4b55b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:21:49.401182 9842 raft_consensus.cc:686] T 00000000000000000000000000000000 P 049b783e402e4560b7d3b9b60de4b55b [term 1 LEADER]: Becoming Leader. State: Replica: 049b783e402e4560b7d3b9b60de4b55b, State: Running, Role: LEADER
I20260928 12:21:49.401221 9839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 049b783e402e4560b7d3b9b60de4b55b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:21:49.401383 9842 consensus_queue.cc:237] T 00000000000000000000000000000000 P 049b783e402e4560b7d3b9b60de4b55b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "049b783e402e4560b7d3b9b60de4b55b" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 42227 } }
I20260928 12:21:49.401858 9844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 049b783e402e4560b7d3b9b60de4b55b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 049b783e402e4560b7d3b9b60de4b55b. Latest consensus state: current_term: 1 leader_uuid: "049b783e402e4560b7d3b9b60de4b55b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "049b783e402e4560b7d3b9b60de4b55b" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 42227 } } }
I20260928 12:21:49.401937 9844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 049b783e402e4560b7d3b9b60de4b55b [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:49.402119 9843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 049b783e402e4560b7d3b9b60de4b55b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "049b783e402e4560b7d3b9b60de4b55b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "049b783e402e4560b7d3b9b60de4b55b" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 42227 } } }
I20260928 12:21:49.402180 9843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 049b783e402e4560b7d3b9b60de4b55b [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:49.402487 9851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:21:49.402992 9851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:21:49.403111 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 9761
I20260928 12:21:49.403192 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:21:49.403393 2272 external_mini_cluster.cc:1428] Setting key bd5ca623a8dd9e91fca2deafed3c26c2
I20260928 12:21:49.404631 9851 catalog_manager.cc:1347] Generated new cluster ID: c4f284c4f0e24b889dbc8e10b15712e2
I20260928 12:21:49.404677 9851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:21:49 dist-test-slave-920p krb5kdc[9748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM
I20260928 12:21:49.415797 9861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:49.403898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35070 (local address 127.2.56.62:42227)
0928 12:21:49.404071 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:49.404081 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:49.404272 (+ 191us) server_negotiation.cc:407] Connection header received
0928 12:21:49.404508 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:49.404517 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:49.404583 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:49.404734 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:49.405686 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.407420 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.408565 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.409203 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.409342 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:49.412211 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:49.412216 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:49.412216 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:49.412229 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:49.414163 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:49.414520 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:49.414524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:49.414524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:49.414635 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:49.414802 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:49.414803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:49.414803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:49.415142 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:49.415309 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:49.415378 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 12:21:49.415533 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
I20260928 12:21:49.421455 9851 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:21:49.422223 9851 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:21:49.425726 9851 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 049b783e402e4560b7d3b9b60de4b55b: Generated new TSK 0
I20260928 12:21:49.425899 9851 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.1.keytab.
Entry for principal oryx/127.2.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.1.keytab.
I20260928 12:21:49.450115 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.1:0
--local_ip_for_outbound_sockets=127.2.56.1
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:42227
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.1.keytab
--principal=oryx/127.2.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:41561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:49.536499 9865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:49.536723 9865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:49.536746 9865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:49.537899 9865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:49.537945 9865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:49.537964 9865 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 12:21:49.537983 9865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.1
I20260928 12:21:49.539427 9865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:41561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.1.keytab
--principal=oryx/127.2.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.56.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.56.62:42227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.1
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:49.539619 9865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:49.539808 9865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:49.544845 9870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:49.544857 9871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:49.544845 9873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:49.545127 9865 server_base.cc:1034] running on GCE node
I20260928 12:21:49.545385 9865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:49.545639 9865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:49.546779 9865 hybrid_clock.cc:648] HybridClock initialized: now 1790598109546769 us; error 30 us; skew 500 ppm
Sep 28 12:21:49 dist-test-slave-920p krb5kdc[9748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598109, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:49.549336 9865 init.cc:377] Logged in from keytab as oryx/127.2.56.1@KRBTEST.COM (short username oryx)
I20260928 12:21:49.549774 9865 webserver.cc:460] Webserver started at http://127.2.56.1:38763/ using document root <none> and password file <none>
I20260928 12:21:49.549993 9865 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:49.550040 9865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:49.550171 9865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:49.551307 9865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-0/data/instance:
uuid: "e016b1c4cd81472c8432f8f34766874f"
format_stamp: "Formatted at 2026-09-28 12:21:49 on dist-test-slave-920p"
server_key: "02dc0ca33ca809fe88fa90049489ee97"
server_key_iv: "a630dd3d8403a4a6fb32691d1d5126d9"
server_key_version: "encryptionkey@0"
I20260928 12:21:49.551656 9865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance:
uuid: "e016b1c4cd81472c8432f8f34766874f"
format_stamp: "Formatted at 2026-09-28 12:21:49 on dist-test-slave-920p"
server_key: "02dc0ca33ca809fe88fa90049489ee97"
server_key_iv: "a630dd3d8403a4a6fb32691d1d5126d9"
server_key_version: "encryptionkey@0"
I20260928 12:21:49.552917 9865 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:21:49.553617 9880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:49.553750 9865 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:49.553833 9865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-0/wal
uuid: "e016b1c4cd81472c8432f8f34766874f"
format_stamp: "Formatted at 2026-09-28 12:21:49 on dist-test-slave-920p"
server_key: "02dc0ca33ca809fe88fa90049489ee97"
server_key_iv: "a630dd3d8403a4a6fb32691d1d5126d9"
server_key_version: "encryptionkey@0"
I20260928 12:21:49.553896 9865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:49.570362 9865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:49.572144 9865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:49.572254 9865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:49.572749 9865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:49.572788 9865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:49.572817 9865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:49.572834 9865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 12:21:49 dist-test-slave-920p krb5kdc[9748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598109, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM
I20260928 12:21:49.580384 9865 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.1:44449
I20260928 12:21:49.580518 9995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.1:44449 every 8 connection(s)
I20260928 12:21:49.580811 9865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
I20260928 12:21:49.582742 9861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:49.573196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:52783 (local address 127.2.56.62:42227)
0928 12:21:49.573242 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:49.573245 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:49.573492 (+ 247us) server_negotiation.cc:407] Connection header received
0928 12:21:49.574075 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:49.574077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:49.574097 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:49.574145 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:49.574848 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.575713 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.576774 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.577104 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.577180 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:49.580132 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:49.580134 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:49.580134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:49.580141 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:49.581650 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:49.581992 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:49.581992 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:49.581993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:49.582043 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:49.582211 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:49.582211 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:49.582211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:49.582317 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:49.582361 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:49.582588 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 12:21:49.582661 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:21:49.582894 9889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:49.573201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:42227 (local address 127.2.56.1:52783)
0928 12:21:49.573371 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:49.573407 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:49.573973 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:49.574232 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:49.574237 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:49.574552 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:49.574757 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:49.574763 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.575809 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.575810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:49.576707 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:49.576710 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.577184 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.577185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:49.577353 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:49.577819 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:49.577829 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:49.580040 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:49.581711 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:49.581715 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:49.581716 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:49.581934 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:49.582086 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:49.582086 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:49.582086 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:49.582172 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:49.582356 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:49.582360 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:49.582485 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 12:21:49.582567 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 12:21:49.582697 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":104,"threads_started":1}
I20260928 12:21:49.583437 9996 heartbeater.cc:347] Connected to a master server at 127.2.56.62:42227
I20260928 12:21:49.583521 9996 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:49.583727 9996 heartbeater.cc:511] Master 127.2.56.62:42227 requested a full tablet report, sending...
I20260928 12:21:49.584303 9792 ts_manager.cc:194] Registered new tserver with Master: e016b1c4cd81472c8432f8f34766874f (127.2.56.1:44449)
I20260928 12:21:49.585045 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 9865
I20260928 12:21:49.585144 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance
I20260928 12:21:49.585235 9792 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.56.1@KRBTEST.COM'} at 127.2.56.1:52783
I20260928 12:21:49.585368 2272 external_mini_cluster.cc:1428] Setting key 28f62689168223d4a2d0ba2ebea3c4bd
I20260928 12:21:49.590023 9861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:49.584563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:35223 (local address 127.2.56.62:42227)
0928 12:21:49.584607 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:49.584609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:49.584678 (+ 69us) server_negotiation.cc:407] Connection header received
0928 12:21:49.584749 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:49.584750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:49.584758 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:49.584799 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:49.585143 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.585765 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.586661 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.586872 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.586937 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:49.587861 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:49.587862 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:49.587862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:49.587868 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:49.589060 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:49.589422 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:49.589422 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:49.589423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:49.589470 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:49.589627 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:49.589627 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:49.589628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:49.589719 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:49.589761 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:49.589895 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 12:21:49.589958 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:21:49.590088 10002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:49.584487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:42227 (local address 127.2.56.1:35223)
0928 12:21:49.584616 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:49.584628 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:49.584694 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:49.584804 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:49.584804 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:49.584997 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:49.585065 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:49.585066 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.585932 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.585933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:49.586588 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:49.586591 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.586936 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.586936 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:49.587006 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:49.587441 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:49.587445 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:49.587791 (+ 346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:49.589127 (+ 1336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:49.589128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:49.589129 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:49.589355 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:49.589517 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:49.589518 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:49.589518 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:49.589578 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:49.589762 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:49.589763 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:49.589809 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 12:21:49.589879 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 12:21:49.589985 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":69,"threads_started":1}
WARNING: no policy specified for oryx/127.2.56.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.56.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.2.keytab.
Entry for principal oryx/127.2.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.2.keytab.
I20260928 12:21:49.609736 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.2:0
--local_ip_for_outbound_sockets=127.2.56.2
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:42227
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.2.keytab
--principal=oryx/127.2.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:41561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:49.697358 10005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:49.697588 10005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:49.697611 10005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:49.698829 10005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:49.698871 10005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:49.698894 10005 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 12:21:49.698923 10005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.2
I20260928 12:21:49.700433 10005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:41561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.2.keytab
--principal=oryx/127.2.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.56.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.56.62:42227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.2
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:49.700667 10005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:49.700933 10005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:49.706327 10013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:49.706468 10005 server_base.cc:1034] running on GCE node
W20260928 12:21:49.706336 10011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:49.706315 10010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:49.706772 10005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:49.707026 10005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:49.708165 10005 hybrid_clock.cc:648] HybridClock initialized: now 1790598109708153 us; error 28 us; skew 500 ppm
Sep 28 12:21:49 dist-test-slave-920p krb5kdc[9748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598109, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:49.710686 10005 init.cc:377] Logged in from keytab as oryx/127.2.56.2@KRBTEST.COM (short username oryx)
I20260928 12:21:49.711153 10005 webserver.cc:460] Webserver started at http://127.2.56.2:37387/ using document root <none> and password file <none>
I20260928 12:21:49.711359 10005 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:49.711413 10005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:49.711544 10005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:49.712672 10005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-1/data/instance:
uuid: "cd35f7e208354f2f88192ad4c475b766"
format_stamp: "Formatted at 2026-09-28 12:21:49 on dist-test-slave-920p"
server_key: "321c64d939e3a1bb76265c8bdbb5422c"
server_key_iv: "ce1594b03fa4130656dd171926c96838"
server_key_version: "encryptionkey@0"
I20260928 12:21:49.713019 10005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance:
uuid: "cd35f7e208354f2f88192ad4c475b766"
format_stamp: "Formatted at 2026-09-28 12:21:49 on dist-test-slave-920p"
server_key: "321c64d939e3a1bb76265c8bdbb5422c"
server_key_iv: "ce1594b03fa4130656dd171926c96838"
server_key_version: "encryptionkey@0"
I20260928 12:21:49.714376 10005 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 12:21:49.715111 10020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:49.715269 10005 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:49.715351 10005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-1/wal
uuid: "cd35f7e208354f2f88192ad4c475b766"
format_stamp: "Formatted at 2026-09-28 12:21:49 on dist-test-slave-920p"
server_key: "321c64d939e3a1bb76265c8bdbb5422c"
server_key_iv: "ce1594b03fa4130656dd171926c96838"
server_key_version: "encryptionkey@0"
I20260928 12:21:49.715430 10005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:49.729717 10005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:49.731647 10005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:49.731793 10005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:49.732386 10005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:49.732432 10005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:49.732463 10005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:49.732486 10005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 12:21:49 dist-test-slave-920p krb5kdc[9748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598109, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM
I20260928 12:21:49.740072 10005 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.2:35533
I20260928 12:21:49.740154 10135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.2:35533 every 8 connection(s)
I20260928 12:21:49.740530 10005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
I20260928 12:21:49.742565 9861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:49.732831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:50357 (local address 127.2.56.62:42227)
0928 12:21:49.732881 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:49.732883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:49.733126 (+ 243us) server_negotiation.cc:407] Connection header received
0928 12:21:49.733780 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:49.733782 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:49.733797 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:49.733858 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:49.734570 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.735379 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.736590 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.736903 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.736967 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:49.740016 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:49.740018 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:49.740018 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:49.740026 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:49.741488 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:49.741830 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:49.741830 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:49.741831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:49.741880 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:49.742040 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:49.742041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:49.742041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:49.742141 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:49.742186 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:49.742429 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 12:21:49.742492 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 12:21:49.742712 10029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:49.732833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:42227 (local address 127.2.56.2:50357)
0928 12:21:49.732999 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:49.733038 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:49.733679 (+ 641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:49.733942 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:49.733947 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:49.734238 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:49.734481 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:49.734488 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.735483 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.735483 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:49.736494 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:49.736498 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.737043 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.737044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:49.737212 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:49.737801 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:49.737810 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:49.739918 (+ 2108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:49.741555 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:49.741560 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:49.741561 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:49.741770 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:49.741918 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:49.741918 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:49.741919 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:49.742002 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:49.742185 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:49.742189 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:49.742309 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 12:21:49.742391 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 12:21:49.742528 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":87,"threads_started":1}
I20260928 12:21:49.743196 10136 heartbeater.cc:347] Connected to a master server at 127.2.56.62:42227
I20260928 12:21:49.743278 10136 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:49.743458 10136 heartbeater.cc:511] Master 127.2.56.62:42227 requested a full tablet report, sending...
I20260928 12:21:49.743866 9793 ts_manager.cc:194] Registered new tserver with Master: cd35f7e208354f2f88192ad4c475b766 (127.2.56.2:35533)
I20260928 12:21:49.744357 9793 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.56.2@KRBTEST.COM'} at 127.2.56.2:50357
I20260928 12:21:49.744740 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 10005
I20260928 12:21:49.744843 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance
I20260928 12:21:49.745064 2272 external_mini_cluster.cc:1428] Setting key 18364ef313c98b915c0c76a1f19f6806
I20260928 12:21:49.749722 9861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:49.744244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:44367 (local address 127.2.56.62:42227)
0928 12:21:49.744284 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:49.744286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:49.744349 (+ 63us) server_negotiation.cc:407] Connection header received
0928 12:21:49.744392 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:49.744393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:49.744403 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:49.744464 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:49.744881 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.745563 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.746287 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.746551 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.746613 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:49.747481 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:49.747482 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:49.747483 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:49.747488 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:49.748664 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:49.748998 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:49.748999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:49.748999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:49.749040 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:49.749183 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:49.749183 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:49.749184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:49.749374 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:49.749436 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:49.749590 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 12:21:49.749656 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 12:21:49.749768 10142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:49.744171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:42227 (local address 127.2.56.2:44367)
0928 12:21:49.744294 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:49.744303 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:49.744356 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:49.744533 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:49.744535 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:49.744738 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:49.744816 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:49.744817 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.745719 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.745720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:49.746217 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:49.746218 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.746612 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.746613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:49.746656 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:49.747031 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:49.747035 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:49.747394 (+ 359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:49.748732 (+ 1338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:49.748733 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:49.748734 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:49.748938 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:49.749080 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:49.749081 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:49.749081 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:49.749136 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:49.749430 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:49.749431 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:49.749478 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 12:21:49.749553 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 12:21:49.749662 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":74,"threads_started":1}
WARNING: no policy specified for oryx/127.2.56.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.56.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.3.keytab.
Entry for principal oryx/127.2.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.3.keytab.
I20260928 12:21:49.768833 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.3:0
--local_ip_for_outbound_sockets=127.2.56.3
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:42227
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.3.keytab
--principal=oryx/127.2.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:41561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:49.856318 10145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:49.856566 10145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:49.856591 10145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:49.857894 10145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:49.857936 10145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:49.857957 10145 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 12:21:49.857990 10145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.3
I20260928 12:21:49.859561 10145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:41561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.3.keytab
--principal=oryx/127.2.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.56.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.56.62:42227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.3
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:49.859815 10145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:49.860054 10145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:49.865219 10153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:49.865283 10151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:49.865283 10150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:49.865509 10145 server_base.cc:1034] running on GCE node
I20260928 12:21:49.865702 10145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:49.865942 10145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:49.867079 10145 hybrid_clock.cc:648] HybridClock initialized: now 1790598109867063 us; error 24 us; skew 500 ppm
Sep 28 12:21:49 dist-test-slave-920p krb5kdc[9748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598109, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:49.869645 10145 init.cc:377] Logged in from keytab as oryx/127.2.56.3@KRBTEST.COM (short username oryx)
I20260928 12:21:49.870112 10145 webserver.cc:460] Webserver started at http://127.2.56.3:35203/ using document root <none> and password file <none>
I20260928 12:21:49.870325 10145 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:49.870373 10145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:49.870502 10145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:49.871628 10145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-2/data/instance:
uuid: "239e6f2672f64a35b0fa47287b25a451"
format_stamp: "Formatted at 2026-09-28 12:21:49 on dist-test-slave-920p"
server_key: "8303d227eab7ff8ed0e3c05284529697"
server_key_iv: "ca19cbea6e2b2a08ad88e1d28c046885"
server_key_version: "encryptionkey@0"
I20260928 12:21:49.871981 10145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance:
uuid: "239e6f2672f64a35b0fa47287b25a451"
format_stamp: "Formatted at 2026-09-28 12:21:49 on dist-test-slave-920p"
server_key: "8303d227eab7ff8ed0e3c05284529697"
server_key_iv: "ca19cbea6e2b2a08ad88e1d28c046885"
server_key_version: "encryptionkey@0"
I20260928 12:21:49.873288 10145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260928 12:21:49.873998 10160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:49.874127 10145 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:21:49.874218 10145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-2/wal
uuid: "239e6f2672f64a35b0fa47287b25a451"
format_stamp: "Formatted at 2026-09-28 12:21:49 on dist-test-slave-920p"
server_key: "8303d227eab7ff8ed0e3c05284529697"
server_key_iv: "ca19cbea6e2b2a08ad88e1d28c046885"
server_key_version: "encryptionkey@0"
I20260928 12:21:49.874275 10145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:49.897338 10145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:49.899750 10145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:49.899899 10145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:49.900498 10145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:49.900539 10145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:49.900574 10145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:49.900599 10145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 12:21:49 dist-test-slave-920p krb5kdc[9748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598109, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM
I20260928 12:21:49.908417 10145 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.3:44701
I20260928 12:21:49.908723 10275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.3:44701 every 8 connection(s)
I20260928 12:21:49.908878 10145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
I20260928 12:21:49.910732 9861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:49.901055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:34083 (local address 127.2.56.62:42227)
0928 12:21:49.901121 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:49.901123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:49.901380 (+ 257us) server_negotiation.cc:407] Connection header received
0928 12:21:49.902077 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:49.902078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:49.902097 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:49.902179 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:49.903148 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.903942 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.905020 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.905343 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.905415 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:49.908160 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:49.908162 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:49.908162 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:49.908171 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:49.909657 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:49.910016 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:49.910017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:49.910017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:49.910063 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:49.910226 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:49.910226 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:49.910226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:49.910326 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:49.910371 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:49.910596 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 12:21:49.910659 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 12:21:49.910928 10169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:49.901063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:42227 (local address 127.2.56.3:34083)
0928 12:21:49.901219 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:49.901282 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:49.901910 (+ 628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:49.902275 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:49.902280 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:49.902711 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:49.903037 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:49.903044 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.904041 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.904042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:49.904933 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:49.904936 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.905432 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.905433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:49.905568 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:49.906142 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:49.906153 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:49.908066 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:49.909722 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:49.909725 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:49.909728 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:49.909955 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:49.910098 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:49.910099 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:49.910099 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:49.910186 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:49.910368 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:49.910371 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:49.910496 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 12:21:49.910577 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 12:21:49.910712 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
I20260928 12:21:49.911374 10276 heartbeater.cc:347] Connected to a master server at 127.2.56.62:42227
I20260928 12:21:49.911458 10276 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:49.911641 10276 heartbeater.cc:511] Master 127.2.56.62:42227 requested a full tablet report, sending...
I20260928 12:21:49.912078 9792 ts_manager.cc:194] Registered new tserver with Master: 239e6f2672f64a35b0fa47287b25a451 (127.2.56.3:44701)
I20260928 12:21:49.912523 9792 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.56.3@KRBTEST.COM'} at 127.2.56.3:34083
I20260928 12:21:49.913689 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 10145
I20260928 12:21:49.913808 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance
I20260928 12:21:49.914024 2272 external_mini_cluster.cc:1428] Setting key a929f80dc09dd5a4fac9ea78ae78bcbd
I20260928 12:21:49.914875 2272 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:21:49.917761 9861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:49.912431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:35547 (local address 127.2.56.62:42227)
0928 12:21:49.912476 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:49.912477 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:49.912585 (+ 108us) server_negotiation.cc:407] Connection header received
0928 12:21:49.912681 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:49.912682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:49.912694 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:49.912752 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:49.913081 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.913601 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.914425 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.914694 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.914754 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:49.915618 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:49.915619 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:49.915619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:49.915625 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:49.916789 (+ 1164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:49.917079 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:49.917080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:49.917080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:49.917130 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:49.917306 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:49.917307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:49.917308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:49.917411 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:49.917465 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:49.917596 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 12:21:49.917667 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 12:21:49.917775 10282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:49.912358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:42227 (local address 127.2.56.3:35547)
0928 12:21:49.912522 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:49.912532 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:49.912608 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:49.912756 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:49.912757 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:49.912929 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:49.913007 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:49.913009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.913686 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.913687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:49.914357 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:49.914359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.914760 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.914761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:49.914834 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:49.915186 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:49.915191 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:49.915535 (+ 344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:49.916862 (+ 1327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:49.916863 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:49.916864 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:49.917020 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:49.917180 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:49.917180 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:49.917181 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:49.917236 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:49.917464 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:49.917465 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:49.917509 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 12:21:49.917569 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 12:21:49.917669 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":69,"threads_started":1}
Sep 28 12:21:49 dist-test-slave-920p krb5kdc[9748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598109, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 12:21:49 dist-test-slave-920p krb5kdc[9748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:21:49 dist-test-slave-920p krb5kdc[9748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 12:21:49.936587 9861 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:21:49.931137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35080 (local address 127.2.56.62:42227)
0928 12:21:49.931201 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:49.931203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:49.931227 (+ 24us) server_negotiation.cc:407] Connection header received
0928 12:21:49.931257 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:49.931258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:49.931269 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:49.931311 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:49.931880 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.932397 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.933171 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.933393 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.933457 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:49.936489 (+ 3032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35080: BlockingRecv error: recv got EOF from 127.0.0.1:35080 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 28 12:21:49 dist-test-slave-920p krb5kdc[9748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598109, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM
I20260928 12:21:49.943998 9861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:49.937329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35082 (local address 127.2.56.62:42227)
0928 12:21:49.937372 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:49.937374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:49.937397 (+ 23us) server_negotiation.cc:407] Connection header received
0928 12:21:49.937459 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:49.937460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:49.937469 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:49.937518 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:49.938039 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.938539 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.939231 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.939416 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.939460 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:49.941668 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:49.941670 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:49.941670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:49.941677 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:49.943000 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:49.943375 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:49.943376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:49.943376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:49.943433 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:49.943585 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:49.943585 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:49.943586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:49.943697 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:49.943781 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:49.943844 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 12:21:49.943907 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:21:49.945711 9792 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:21:49.947613 9792 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:21:49.954721 10298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:49.950123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:35533 (local address 127.0.0.1:56496)
0928 12:21:49.950620 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:49.950631 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:49.950711 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:49.950945 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:49.950946 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:49.950948 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:49.950996 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:49.950997 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.952292 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.952293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:49.953441 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:49.953444 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.954380 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.954381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:49.954470 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:49.954488 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 12:21:49.954572 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 12:21:49.954632 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":33,"thread_start_us":117,"threads_started":1}
I20260928 12:21:49.954748 10301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:49.950426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56496 (local address 127.2.56.2:35533)
0928 12:21:49.950604 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:49.950610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:49.950713 (+ 103us) server_negotiation.cc:407] Connection header received
0928 12:21:49.950819 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:49.950822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:49.950837 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:49.951091 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:49.951166 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.952215 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.953518 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.954297 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.954371 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:49.954467 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:49.954594 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 12:21:49.954632 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":97,"threads_started":1}
I20260928 12:21:49.955497 10300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:49.950104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:44701 (local address 127.0.0.1:42046)
0928 12:21:49.950288 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:49.950335 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:49.950408 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:49.950795 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:49.950798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:49.950802 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:49.950895 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:49.950898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.953318 (+ 2420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.953319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:49.954352 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:49.954355 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.955283 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.955284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:49.955358 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:49.955362 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:21:49.955391 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 12:21:49.955432 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":70,"threads_started":1}
I20260928 12:21:49.955711 10299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:49.950123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:44449 (local address 127.0.0.1:58382)
0928 12:21:49.950288 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:49.950335 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:49.950421 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:49.951162 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:49.951163 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:49.951165 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:49.951211 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:49.951213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.952070 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.952072 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:49.953441 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:49.953444 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.955493 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.955494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:49.955558 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:49.955563 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:21:49.955605 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 12:21:49.955647 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":2,"thread_start_us":106,"threads_started":1}
I20260928 12:21:49.955775 10303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:49.950666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58382 (local address 127.2.56.1:44449)
0928 12:21:49.951013 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:49.951019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:49.951028 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:21:49.951067 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:49.951069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:49.951082 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:49.951155 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:49.951272 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.951994 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.954508 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.955407 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.955470 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:49.955552 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:49.955628 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 12:21:49.955675 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":106,"threads_started":1}
I20260928 12:21:49.955921 10302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:49.950416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42046 (local address 127.2.56.3:44701)
0928 12:21:49.950604 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:49.950610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:49.950620 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:21:49.950662 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:49.950665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:49.950682 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:49.950972 (+ 290us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:49.951040 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.952205 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.954432 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.955188 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.955643 (+ 455us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:49.955738 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:49.955763 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 12:21:49.955812 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":112,"threads_started":1}
I20260928 12:21:49.955780 10070 tablet_service.cc:1467] Processing CreateTablet for tablet 920621fc93dd4e18bb4bd3285ddaeda9 (DEFAULT_TABLE table=test-table [id=bcd11d4e79e542e29ac05925d3be2330]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:49.956240 10070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 920621fc93dd4e18bb4bd3285ddaeda9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:49.956460 10210 tablet_service.cc:1467] Processing CreateTablet for tablet 920621fc93dd4e18bb4bd3285ddaeda9 (DEFAULT_TABLE table=test-table [id=bcd11d4e79e542e29ac05925d3be2330]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:49.956463 9930 tablet_service.cc:1467] Processing CreateTablet for tablet 920621fc93dd4e18bb4bd3285ddaeda9 (DEFAULT_TABLE table=test-table [id=bcd11d4e79e542e29ac05925d3be2330]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:49.956734 10210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 920621fc93dd4e18bb4bd3285ddaeda9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:49.956817 9930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 920621fc93dd4e18bb4bd3285ddaeda9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:49.960089 10304 tablet_bootstrap.cc:492] T 920621fc93dd4e18bb4bd3285ddaeda9 P 239e6f2672f64a35b0fa47287b25a451: Bootstrap starting.
I20260928 12:21:49.960498 10305 tablet_bootstrap.cc:492] T 920621fc93dd4e18bb4bd3285ddaeda9 P cd35f7e208354f2f88192ad4c475b766: Bootstrap starting.
I20260928 12:21:49.960717 10306 tablet_bootstrap.cc:492] T 920621fc93dd4e18bb4bd3285ddaeda9 P e016b1c4cd81472c8432f8f34766874f: Bootstrap starting.
I20260928 12:21:49.960780 10304 tablet_bootstrap.cc:654] T 920621fc93dd4e18bb4bd3285ddaeda9 P 239e6f2672f64a35b0fa47287b25a451: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:49.961133 10304 log.cc:824] T 920621fc93dd4e18bb4bd3285ddaeda9 P 239e6f2672f64a35b0fa47287b25a451: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:49.961184 10305 tablet_bootstrap.cc:654] T 920621fc93dd4e18bb4bd3285ddaeda9 P cd35f7e208354f2f88192ad4c475b766: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:49.961364 10306 tablet_bootstrap.cc:654] T 920621fc93dd4e18bb4bd3285ddaeda9 P e016b1c4cd81472c8432f8f34766874f: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:49.961723 10305 log.cc:824] T 920621fc93dd4e18bb4bd3285ddaeda9 P cd35f7e208354f2f88192ad4c475b766: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:49.961727 10306 log.cc:824] T 920621fc93dd4e18bb4bd3285ddaeda9 P e016b1c4cd81472c8432f8f34766874f: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:49.962105 10304 tablet_bootstrap.cc:492] T 920621fc93dd4e18bb4bd3285ddaeda9 P 239e6f2672f64a35b0fa47287b25a451: No bootstrap required, opened a new log
I20260928 12:21:49.962172 10304 ts_tablet_manager.cc:1397] T 920621fc93dd4e18bb4bd3285ddaeda9 P 239e6f2672f64a35b0fa47287b25a451: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:21:49.962447 10305 tablet_bootstrap.cc:492] T 920621fc93dd4e18bb4bd3285ddaeda9 P cd35f7e208354f2f88192ad4c475b766: No bootstrap required, opened a new log
I20260928 12:21:49.962517 10305 ts_tablet_manager.cc:1397] T 920621fc93dd4e18bb4bd3285ddaeda9 P cd35f7e208354f2f88192ad4c475b766: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:21:49.964196 10304 raft_consensus.cc:348] T 920621fc93dd4e18bb4bd3285ddaeda9 P 239e6f2672f64a35b0fa47287b25a451 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } } peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } }
I20260928 12:21:49.964388 10304 raft_consensus.cc:374] T 920621fc93dd4e18bb4bd3285ddaeda9 P 239e6f2672f64a35b0fa47287b25a451 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:49.964435 10304 raft_consensus.cc:729] T 920621fc93dd4e18bb4bd3285ddaeda9 P 239e6f2672f64a35b0fa47287b25a451 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 239e6f2672f64a35b0fa47287b25a451, State: Initialized, Role: FOLLOWER
I20260928 12:21:49.964484 10305 raft_consensus.cc:348] T 920621fc93dd4e18bb4bd3285ddaeda9 P cd35f7e208354f2f88192ad4c475b766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } } peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } }
I20260928 12:21:49.964568 10304 consensus_queue.cc:260] T 920621fc93dd4e18bb4bd3285ddaeda9 P 239e6f2672f64a35b0fa47287b25a451 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } } peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } }
I20260928 12:21:49.964632 10305 raft_consensus.cc:374] T 920621fc93dd4e18bb4bd3285ddaeda9 P cd35f7e208354f2f88192ad4c475b766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:49.964668 10305 raft_consensus.cc:729] T 920621fc93dd4e18bb4bd3285ddaeda9 P cd35f7e208354f2f88192ad4c475b766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd35f7e208354f2f88192ad4c475b766, State: Initialized, Role: FOLLOWER
I20260928 12:21:49.964778 10305 consensus_queue.cc:260] T 920621fc93dd4e18bb4bd3285ddaeda9 P cd35f7e208354f2f88192ad4c475b766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } } peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } }
I20260928 12:21:49.964852 10304 ts_tablet_manager.cc:1428] T 920621fc93dd4e18bb4bd3285ddaeda9 P 239e6f2672f64a35b0fa47287b25a451: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:21:49.964905 10276 heartbeater.cc:503] Master 127.2.56.62:42227 was elected leader, sending a full tablet report...
I20260928 12:21:49.965024 10305 ts_tablet_manager.cc:1428] T 920621fc93dd4e18bb4bd3285ddaeda9 P cd35f7e208354f2f88192ad4c475b766: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 12:21:49.965137 10136 heartbeater.cc:503] Master 127.2.56.62:42227 was elected leader, sending a full tablet report...
I20260928 12:21:49.965536 10306 tablet_bootstrap.cc:492] T 920621fc93dd4e18bb4bd3285ddaeda9 P e016b1c4cd81472c8432f8f34766874f: No bootstrap required, opened a new log
I20260928 12:21:49.965608 10306 ts_tablet_manager.cc:1397] T 920621fc93dd4e18bb4bd3285ddaeda9 P e016b1c4cd81472c8432f8f34766874f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260928 12:21:49.967212 10306 raft_consensus.cc:348] T 920621fc93dd4e18bb4bd3285ddaeda9 P e016b1c4cd81472c8432f8f34766874f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } } peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } }
I20260928 12:21:49.967350 10306 raft_consensus.cc:374] T 920621fc93dd4e18bb4bd3285ddaeda9 P e016b1c4cd81472c8432f8f34766874f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:49.967379 10306 raft_consensus.cc:729] T 920621fc93dd4e18bb4bd3285ddaeda9 P e016b1c4cd81472c8432f8f34766874f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e016b1c4cd81472c8432f8f34766874f, State: Initialized, Role: FOLLOWER
I20260928 12:21:49.967501 10306 consensus_queue.cc:260] T 920621fc93dd4e18bb4bd3285ddaeda9 P e016b1c4cd81472c8432f8f34766874f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } } peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } }
I20260928 12:21:49.967765 10306 ts_tablet_manager.cc:1428] T 920621fc93dd4e18bb4bd3285ddaeda9 P e016b1c4cd81472c8432f8f34766874f: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260928 12:21:49.967828 9996 heartbeater.cc:503] Master 127.2.56.62:42227 was elected leader, sending a full tablet report...
I20260928 12:21:49.991117 10310 raft_consensus.cc:482] T 920621fc93dd4e18bb4bd3285ddaeda9 P 239e6f2672f64a35b0fa47287b25a451 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:21:49.991271 10310 raft_consensus.cc:504] T 920621fc93dd4e18bb4bd3285ddaeda9 P 239e6f2672f64a35b0fa47287b25a451 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } } peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } }
W20260928 12:21:49.991324 10137 tablet.cc:2369] T 920621fc93dd4e18bb4bd3285ddaeda9 P cd35f7e208354f2f88192ad4c475b766: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:21:49.991717 10310 leader_election.cc:288] T 920621fc93dd4e18bb4bd3285ddaeda9 P 239e6f2672f64a35b0fa47287b25a451 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e016b1c4cd81472c8432f8f34766874f (127.2.56.1:44449), cd35f7e208354f2f88192ad4c475b766 (127.2.56.2:35533)
I20260928 12:21:49.995604 10303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:49.991980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:56909 (local address 127.2.56.1:44449)
0928 12:21:49.992036 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:49.992038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:49.992048 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:21:49.992097 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:49.992098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:49.992112 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:49.992157 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:49.992356 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.993145 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.994665 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.995251 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.995322 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:49.995335 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:49.995490 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 12:21:49.995527 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 12:21:49.995604 10301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:49.992116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:59747 (local address 127.2.56.2:35533)
0928 12:21:49.992165 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:49.992167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:49.992187 (+ 20us) server_negotiation.cc:407] Connection header received
0928 12:21:49.992241 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:49.992242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:49.992253 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:49.992294 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:49.992434 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.993148 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.994664 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.995249 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.995322 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:49.995335 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:49.995490 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 12:21:49.995527 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 12:21:49.995627 10169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:49.991914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:44449 (local address 127.2.56.3:56909)
0928 12:21:49.991976 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:49.991989 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:49.992034 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:49.992198 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:49.992199 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:49.992201 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:49.992283 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:49.992285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.993223 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.993223 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:49.994569 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:49.994572 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.995322 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.995323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:49.995427 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:49.995430 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:21:49.995479 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 12:21:49.995529 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":97}
I20260928 12:21:49.995627 10313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:49.991927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:35533 (local address 127.2.56.3:59747)
0928 12:21:49.992130 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:49.992139 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:49.992185 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:49.992313 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:49.992314 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:49.992315 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:49.992378 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:49.992379 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.993219 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.993220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:49.994569 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:49.994572 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:49.995323 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:49.995324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:49.995419 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:49.995424 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:21:49.995479 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 12:21:49.995529 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":31,"thread_start_us":135,"threads_started":1}
I20260928 12:21:49.996032 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "920621fc93dd4e18bb4bd3285ddaeda9" candidate_uuid: "239e6f2672f64a35b0fa47287b25a451" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd35f7e208354f2f88192ad4c475b766" is_pre_election: true
I20260928 12:21:49.996050 9950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "920621fc93dd4e18bb4bd3285ddaeda9" candidate_uuid: "239e6f2672f64a35b0fa47287b25a451" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e016b1c4cd81472c8432f8f34766874f" is_pre_election: true
I20260928 12:21:49.996196 10086 raft_consensus.cc:2454] T 920621fc93dd4e18bb4bd3285ddaeda9 P cd35f7e208354f2f88192ad4c475b766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 239e6f2672f64a35b0fa47287b25a451 in term 0.
I20260928 12:21:49.996196 9950 raft_consensus.cc:2454] T 920621fc93dd4e18bb4bd3285ddaeda9 P e016b1c4cd81472c8432f8f34766874f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 239e6f2672f64a35b0fa47287b25a451 in term 0.
I20260928 12:21:49.996415 10164 leader_election.cc:302] T 920621fc93dd4e18bb4bd3285ddaeda9 P 239e6f2672f64a35b0fa47287b25a451 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 239e6f2672f64a35b0fa47287b25a451, cd35f7e208354f2f88192ad4c475b766; no voters:
I20260928 12:21:49.996567 10310 raft_consensus.cc:2781] T 920621fc93dd4e18bb4bd3285ddaeda9 P 239e6f2672f64a35b0fa47287b25a451 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:21:49.996624 10310 raft_consensus.cc:482] T 920621fc93dd4e18bb4bd3285ddaeda9 P 239e6f2672f64a35b0fa47287b25a451 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:21:49.996640 10310 raft_consensus.cc:3037] T 920621fc93dd4e18bb4bd3285ddaeda9 P 239e6f2672f64a35b0fa47287b25a451 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:49.997637 10310 raft_consensus.cc:504] T 920621fc93dd4e18bb4bd3285ddaeda9 P 239e6f2672f64a35b0fa47287b25a451 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } } peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } }
I20260928 12:21:49.997753 10310 leader_election.cc:288] T 920621fc93dd4e18bb4bd3285ddaeda9 P 239e6f2672f64a35b0fa47287b25a451 [CANDIDATE]: Term 1 election: Requested vote from peers e016b1c4cd81472c8432f8f34766874f (127.2.56.1:44449), cd35f7e208354f2f88192ad4c475b766 (127.2.56.2:35533)
I20260928 12:21:49.997888 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "920621fc93dd4e18bb4bd3285ddaeda9" candidate_uuid: "239e6f2672f64a35b0fa47287b25a451" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd35f7e208354f2f88192ad4c475b766"
I20260928 12:21:49.997896 9950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "920621fc93dd4e18bb4bd3285ddaeda9" candidate_uuid: "239e6f2672f64a35b0fa47287b25a451" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e016b1c4cd81472c8432f8f34766874f"
I20260928 12:21:49.997967 10086 raft_consensus.cc:3037] T 920621fc93dd4e18bb4bd3285ddaeda9 P cd35f7e208354f2f88192ad4c475b766 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:49.997972 9950 raft_consensus.cc:3037] T 920621fc93dd4e18bb4bd3285ddaeda9 P e016b1c4cd81472c8432f8f34766874f [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:49.998689 10086 raft_consensus.cc:2454] T 920621fc93dd4e18bb4bd3285ddaeda9 P cd35f7e208354f2f88192ad4c475b766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 239e6f2672f64a35b0fa47287b25a451 in term 1.
I20260928 12:21:49.998694 9950 raft_consensus.cc:2454] T 920621fc93dd4e18bb4bd3285ddaeda9 P e016b1c4cd81472c8432f8f34766874f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 239e6f2672f64a35b0fa47287b25a451 in term 1.
I20260928 12:21:49.998833 10164 leader_election.cc:302] T 920621fc93dd4e18bb4bd3285ddaeda9 P 239e6f2672f64a35b0fa47287b25a451 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 239e6f2672f64a35b0fa47287b25a451, cd35f7e208354f2f88192ad4c475b766; no voters:
I20260928 12:21:49.998909 10310 raft_consensus.cc:2781] T 920621fc93dd4e18bb4bd3285ddaeda9 P 239e6f2672f64a35b0fa47287b25a451 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:21:49.999063 10310 raft_consensus.cc:686] T 920621fc93dd4e18bb4bd3285ddaeda9 P 239e6f2672f64a35b0fa47287b25a451 [term 1 LEADER]: Becoming Leader. State: Replica: 239e6f2672f64a35b0fa47287b25a451, State: Running, Role: LEADER
I20260928 12:21:49.999150 10310 consensus_queue.cc:237] T 920621fc93dd4e18bb4bd3285ddaeda9 P 239e6f2672f64a35b0fa47287b25a451 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } } peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } }
I20260928 12:21:50.000157 9793 catalog_manager.cc:5495] T 920621fc93dd4e18bb4bd3285ddaeda9 P 239e6f2672f64a35b0fa47287b25a451 reported cstate change: term changed from 0 to 1, leader changed from <none> to 239e6f2672f64a35b0fa47287b25a451 (127.2.56.3). New cstate: current_term: 1 leader_uuid: "239e6f2672f64a35b0fa47287b25a451" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } health_report { overall_health: UNKNOWN } } }
I20260928 12:21:50.011399 10302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:50.009131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42052 (local address 127.2.56.3:44701)
0928 12:21:50.009194 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:50.009196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:50.009215 (+ 19us) server_negotiation.cc:407] Connection header received
0928 12:21:50.009253 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:50.009254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:50.009311 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:50.009371 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:21:50.009514 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:50.010004 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:50.010718 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:50.010900 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:50.010948 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:50.011083 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:21:50.011218 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:21:50.011279 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:50.011296 (+ 17us) server_negotiation.cc:299] Negotiation successful
0928 12:21:50.011314 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 12:21:50.013549 10086 raft_consensus.cc:1261] T 920621fc93dd4e18bb4bd3285ddaeda9 P cd35f7e208354f2f88192ad4c475b766 [term 1 FOLLOWER]: Refusing update from remote peer 239e6f2672f64a35b0fa47287b25a451: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:21:50.013579 9950 raft_consensus.cc:1261] T 920621fc93dd4e18bb4bd3285ddaeda9 P e016b1c4cd81472c8432f8f34766874f [term 1 FOLLOWER]: Refusing update from remote peer 239e6f2672f64a35b0fa47287b25a451: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:21:50.013906 10310 consensus_queue.cc:1035] T 920621fc93dd4e18bb4bd3285ddaeda9 P 239e6f2672f64a35b0fa47287b25a451 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:21:50.014081 10314 consensus_queue.cc:1035] T 920621fc93dd4e18bb4bd3285ddaeda9 P 239e6f2672f64a35b0fa47287b25a451 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:21:50.017123 10319 mvcc.cc:205] Tried to move back new op lower bound from 7334289858612498432 to 7334289858557235200. Current Snapshot: MvccSnapshot[applied={T|T < 7334289858612498432 or (T in {7334289858612498432})}]
I20260928 12:21:50.019577 9793 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:21:50.019691 9793 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:21:50.020470 9793 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 049b783e402e4560b7d3b9b60de4b55b: Sending DeleteTablet for 3 replicas of tablet 920621fc93dd4e18bb4bd3285ddaeda9
I20260928 12:21:50.020850 9930 tablet_service.cc:1514] Processing DeleteTablet for tablet 920621fc93dd4e18bb4bd3285ddaeda9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:50 UTC) from {username='oryx', principal='oryx/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:58382
I20260928 12:21:50.020936 10210 tablet_service.cc:1514] Processing DeleteTablet for tablet 920621fc93dd4e18bb4bd3285ddaeda9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:50 UTC) from {username='oryx', principal='oryx/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:42046
I20260928 12:21:50.021013 10070 tablet_service.cc:1514] Processing DeleteTablet for tablet 920621fc93dd4e18bb4bd3285ddaeda9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:50 UTC) from {username='oryx', principal='oryx/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:56496
I20260928 12:21:50.021198 10328 tablet_replica.cc:330] T 920621fc93dd4e18bb4bd3285ddaeda9 P 239e6f2672f64a35b0fa47287b25a451: stopping tablet replica
I20260928 12:21:50.021324 10328 raft_consensus.cc:2229] T 920621fc93dd4e18bb4bd3285ddaeda9 P 239e6f2672f64a35b0fa47287b25a451 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:21:50.021286 9793 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:21:50.021454 9793 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:21:50.021481 10328 raft_consensus.cc:2258] T 920621fc93dd4e18bb4bd3285ddaeda9 P 239e6f2672f64a35b0fa47287b25a451 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:50.021631 10329 tablet_replica.cc:330] T 920621fc93dd4e18bb4bd3285ddaeda9 P cd35f7e208354f2f88192ad4c475b766: stopping tablet replica
I20260928 12:21:50.021733 10329 raft_consensus.cc:2229] T 920621fc93dd4e18bb4bd3285ddaeda9 P cd35f7e208354f2f88192ad4c475b766 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:21:50.021844 10329 raft_consensus.cc:2258] T 920621fc93dd4e18bb4bd3285ddaeda9 P cd35f7e208354f2f88192ad4c475b766 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:50.022166 10328 ts_tablet_manager.cc:1905] T 920621fc93dd4e18bb4bd3285ddaeda9 P 239e6f2672f64a35b0fa47287b25a451: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:50.022264 10329 ts_tablet_manager.cc:1905] T 920621fc93dd4e18bb4bd3285ddaeda9 P cd35f7e208354f2f88192ad4c475b766: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:50.022964 9930 tablet_service.cc:1467] Processing CreateTablet for tablet ccc929ef361e48e58f25eaf639d5b921 (DEFAULT_TABLE table=test-table [id=1c8afd4143b54988aa627895408f3faa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:50.023103 9930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccc929ef361e48e58f25eaf639d5b921. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:50.024120 10329 ts_tablet_manager.cc:1918] T 920621fc93dd4e18bb4bd3285ddaeda9 P cd35f7e208354f2f88192ad4c475b766: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:21:50.024200 10329 log.cc:1196] T 920621fc93dd4e18bb4bd3285ddaeda9 P cd35f7e208354f2f88192ad4c475b766: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-1/wal/wals/920621fc93dd4e18bb4bd3285ddaeda9
I20260928 12:21:50.024158 10328 ts_tablet_manager.cc:1918] T 920621fc93dd4e18bb4bd3285ddaeda9 P 239e6f2672f64a35b0fa47287b25a451: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:21:50.024380 10328 log.cc:1196] T 920621fc93dd4e18bb4bd3285ddaeda9 P 239e6f2672f64a35b0fa47287b25a451: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-2/wal/wals/920621fc93dd4e18bb4bd3285ddaeda9
I20260928 12:21:50.024442 10329 ts_tablet_manager.cc:1939] T 920621fc93dd4e18bb4bd3285ddaeda9 P cd35f7e208354f2f88192ad4c475b766: Deleting consensus metadata
I20260928 12:21:50.024688 10328 ts_tablet_manager.cc:1939] T 920621fc93dd4e18bb4bd3285ddaeda9 P 239e6f2672f64a35b0fa47287b25a451: Deleting consensus metadata
I20260928 12:21:50.024714 10070 tablet_service.cc:1467] Processing CreateTablet for tablet ccc929ef361e48e58f25eaf639d5b921 (DEFAULT_TABLE table=test-table [id=1c8afd4143b54988aa627895408f3faa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:50.024823 10070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccc929ef361e48e58f25eaf639d5b921. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:50.024842 9780 catalog_manager.cc:4841] TS cd35f7e208354f2f88192ad4c475b766 (127.2.56.2:35533): tablet 920621fc93dd4e18bb4bd3285ddaeda9 (table test-table [id=bcd11d4e79e542e29ac05925d3be2330]) successfully deleted
I20260928 12:21:50.026001 10305 tablet_bootstrap.cc:492] T ccc929ef361e48e58f25eaf639d5b921 P cd35f7e208354f2f88192ad4c475b766: Bootstrap starting.
I20260928 12:21:50.026034 10306 tablet_bootstrap.cc:492] T ccc929ef361e48e58f25eaf639d5b921 P e016b1c4cd81472c8432f8f34766874f: Bootstrap starting.
I20260928 12:21:50.026364 10305 tablet_bootstrap.cc:654] T ccc929ef361e48e58f25eaf639d5b921 P cd35f7e208354f2f88192ad4c475b766: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:50.026746 10306 tablet_bootstrap.cc:654] T ccc929ef361e48e58f25eaf639d5b921 P e016b1c4cd81472c8432f8f34766874f: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:50.026922 10305 tablet_bootstrap.cc:492] T ccc929ef361e48e58f25eaf639d5b921 P cd35f7e208354f2f88192ad4c475b766: No bootstrap required, opened a new log
I20260928 12:21:50.026960 10305 ts_tablet_manager.cc:1397] T ccc929ef361e48e58f25eaf639d5b921 P cd35f7e208354f2f88192ad4c475b766: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:21:50.027122 10305 raft_consensus.cc:348] T ccc929ef361e48e58f25eaf639d5b921 P cd35f7e208354f2f88192ad4c475b766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } }
I20260928 12:21:50.027202 10305 raft_consensus.cc:374] T ccc929ef361e48e58f25eaf639d5b921 P cd35f7e208354f2f88192ad4c475b766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:50.027228 10305 raft_consensus.cc:729] T ccc929ef361e48e58f25eaf639d5b921 P cd35f7e208354f2f88192ad4c475b766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd35f7e208354f2f88192ad4c475b766, State: Initialized, Role: FOLLOWER
I20260928 12:21:50.027271 10305 consensus_queue.cc:260] T ccc929ef361e48e58f25eaf639d5b921 P cd35f7e208354f2f88192ad4c475b766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } }
I20260928 12:21:50.027349 10305 ts_tablet_manager.cc:1428] T ccc929ef361e48e58f25eaf639d5b921 P cd35f7e208354f2f88192ad4c475b766: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:50.027377 9779 catalog_manager.cc:4841] TS 239e6f2672f64a35b0fa47287b25a451 (127.2.56.3:44701): tablet 920621fc93dd4e18bb4bd3285ddaeda9 (table test-table [id=bcd11d4e79e542e29ac05925d3be2330]) successfully deleted
I20260928 12:21:50.027580 10210 tablet_service.cc:1467] Processing CreateTablet for tablet ccc929ef361e48e58f25eaf639d5b921 (DEFAULT_TABLE table=test-table [id=1c8afd4143b54988aa627895408f3faa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:50.027674 10210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccc929ef361e48e58f25eaf639d5b921. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:50.027711 10306 tablet_bootstrap.cc:492] T ccc929ef361e48e58f25eaf639d5b921 P e016b1c4cd81472c8432f8f34766874f: No bootstrap required, opened a new log
I20260928 12:21:50.027752 10306 ts_tablet_manager.cc:1397] T ccc929ef361e48e58f25eaf639d5b921 P e016b1c4cd81472c8432f8f34766874f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260928 12:21:50.027911 10306 raft_consensus.cc:348] T ccc929ef361e48e58f25eaf639d5b921 P e016b1c4cd81472c8432f8f34766874f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } }
I20260928 12:21:50.027979 10306 raft_consensus.cc:374] T ccc929ef361e48e58f25eaf639d5b921 P e016b1c4cd81472c8432f8f34766874f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:50.027998 10306 raft_consensus.cc:729] T ccc929ef361e48e58f25eaf639d5b921 P e016b1c4cd81472c8432f8f34766874f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e016b1c4cd81472c8432f8f34766874f, State: Initialized, Role: FOLLOWER
I20260928 12:21:50.028045 10306 consensus_queue.cc:260] T ccc929ef361e48e58f25eaf639d5b921 P e016b1c4cd81472c8432f8f34766874f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } }
I20260928 12:21:50.028129 10306 ts_tablet_manager.cc:1428] T ccc929ef361e48e58f25eaf639d5b921 P e016b1c4cd81472c8432f8f34766874f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:50.028241 10327 tablet_replica.cc:330] T 920621fc93dd4e18bb4bd3285ddaeda9 P e016b1c4cd81472c8432f8f34766874f: stopping tablet replica
I20260928 12:21:50.028342 10327 raft_consensus.cc:2229] T 920621fc93dd4e18bb4bd3285ddaeda9 P e016b1c4cd81472c8432f8f34766874f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:21:50.028456 10327 raft_consensus.cc:2258] T 920621fc93dd4e18bb4bd3285ddaeda9 P e016b1c4cd81472c8432f8f34766874f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:50.028931 10304 tablet_bootstrap.cc:492] T ccc929ef361e48e58f25eaf639d5b921 P 239e6f2672f64a35b0fa47287b25a451: Bootstrap starting.
I20260928 12:21:50.028968 10327 ts_tablet_manager.cc:1905] T 920621fc93dd4e18bb4bd3285ddaeda9 P e016b1c4cd81472c8432f8f34766874f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:50.029381 10304 tablet_bootstrap.cc:654] T ccc929ef361e48e58f25eaf639d5b921 P 239e6f2672f64a35b0fa47287b25a451: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:50.029963 10304 tablet_bootstrap.cc:492] T ccc929ef361e48e58f25eaf639d5b921 P 239e6f2672f64a35b0fa47287b25a451: No bootstrap required, opened a new log
I20260928 12:21:50.030006 10304 ts_tablet_manager.cc:1397] T ccc929ef361e48e58f25eaf639d5b921 P 239e6f2672f64a35b0fa47287b25a451: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 12:21:50.030154 10304 raft_consensus.cc:348] T ccc929ef361e48e58f25eaf639d5b921 P 239e6f2672f64a35b0fa47287b25a451 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } }
I20260928 12:21:50.030207 10304 raft_consensus.cc:374] T ccc929ef361e48e58f25eaf639d5b921 P 239e6f2672f64a35b0fa47287b25a451 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:50.030224 10304 raft_consensus.cc:729] T ccc929ef361e48e58f25eaf639d5b921 P 239e6f2672f64a35b0fa47287b25a451 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 239e6f2672f64a35b0fa47287b25a451, State: Initialized, Role: FOLLOWER
I20260928 12:21:50.030263 10304 consensus_queue.cc:260] T ccc929ef361e48e58f25eaf639d5b921 P 239e6f2672f64a35b0fa47287b25a451 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } }
I20260928 12:21:50.030342 10304 ts_tablet_manager.cc:1428] T ccc929ef361e48e58f25eaf639d5b921 P 239e6f2672f64a35b0fa47287b25a451: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:50.030478 10327 ts_tablet_manager.cc:1918] T 920621fc93dd4e18bb4bd3285ddaeda9 P e016b1c4cd81472c8432f8f34766874f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:21:50.030546 10327 log.cc:1196] T 920621fc93dd4e18bb4bd3285ddaeda9 P e016b1c4cd81472c8432f8f34766874f: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-0/wal/wals/920621fc93dd4e18bb4bd3285ddaeda9
I20260928 12:21:50.030767 10327 ts_tablet_manager.cc:1939] T 920621fc93dd4e18bb4bd3285ddaeda9 P e016b1c4cd81472c8432f8f34766874f: Deleting consensus metadata
I20260928 12:21:50.031133 9778 catalog_manager.cc:4841] TS e016b1c4cd81472c8432f8f34766874f (127.2.56.1:44449): tablet 920621fc93dd4e18bb4bd3285ddaeda9 (table test-table [id=bcd11d4e79e542e29ac05925d3be2330]) successfully deleted
W20260928 12:21:50.081622 9997 tablet.cc:2369] T ccc929ef361e48e58f25eaf639d5b921 P e016b1c4cd81472c8432f8f34766874f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:21:50.159533 10277 tablet.cc:2369] T ccc929ef361e48e58f25eaf639d5b921 P 239e6f2672f64a35b0fa47287b25a451: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:21:50.220031 10314 raft_consensus.cc:482] T ccc929ef361e48e58f25eaf639d5b921 P 239e6f2672f64a35b0fa47287b25a451 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:21:50.220160 10314 raft_consensus.cc:504] T ccc929ef361e48e58f25eaf639d5b921 P 239e6f2672f64a35b0fa47287b25a451 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } }
I20260928 12:21:50.220336 10314 leader_election.cc:288] T ccc929ef361e48e58f25eaf639d5b921 P 239e6f2672f64a35b0fa47287b25a451 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e016b1c4cd81472c8432f8f34766874f (127.2.56.1:44449), cd35f7e208354f2f88192ad4c475b766 (127.2.56.2:35533)
I20260928 12:21:50.220590 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccc929ef361e48e58f25eaf639d5b921" candidate_uuid: "239e6f2672f64a35b0fa47287b25a451" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd35f7e208354f2f88192ad4c475b766" is_pre_election: true
I20260928 12:21:50.220608 9950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccc929ef361e48e58f25eaf639d5b921" candidate_uuid: "239e6f2672f64a35b0fa47287b25a451" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e016b1c4cd81472c8432f8f34766874f" is_pre_election: true
I20260928 12:21:50.220697 10086 raft_consensus.cc:2454] T ccc929ef361e48e58f25eaf639d5b921 P cd35f7e208354f2f88192ad4c475b766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 239e6f2672f64a35b0fa47287b25a451 in term 0.
I20260928 12:21:50.220722 9950 raft_consensus.cc:2454] T ccc929ef361e48e58f25eaf639d5b921 P e016b1c4cd81472c8432f8f34766874f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 239e6f2672f64a35b0fa47287b25a451 in term 0.
I20260928 12:21:50.220907 10162 leader_election.cc:302] T ccc929ef361e48e58f25eaf639d5b921 P 239e6f2672f64a35b0fa47287b25a451 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 239e6f2672f64a35b0fa47287b25a451, e016b1c4cd81472c8432f8f34766874f; no voters:
I20260928 12:21:50.221024 10314 raft_consensus.cc:2781] T ccc929ef361e48e58f25eaf639d5b921 P 239e6f2672f64a35b0fa47287b25a451 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:21:50.221064 10314 raft_consensus.cc:482] T ccc929ef361e48e58f25eaf639d5b921 P 239e6f2672f64a35b0fa47287b25a451 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:21:50.221078 10314 raft_consensus.cc:3037] T ccc929ef361e48e58f25eaf639d5b921 P 239e6f2672f64a35b0fa47287b25a451 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:50.221992 10314 raft_consensus.cc:504] T ccc929ef361e48e58f25eaf639d5b921 P 239e6f2672f64a35b0fa47287b25a451 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } }
I20260928 12:21:50.222131 10314 leader_election.cc:288] T ccc929ef361e48e58f25eaf639d5b921 P 239e6f2672f64a35b0fa47287b25a451 [CANDIDATE]: Term 1 election: Requested vote from peers e016b1c4cd81472c8432f8f34766874f (127.2.56.1:44449), cd35f7e208354f2f88192ad4c475b766 (127.2.56.2:35533)
I20260928 12:21:50.222268 9950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccc929ef361e48e58f25eaf639d5b921" candidate_uuid: "239e6f2672f64a35b0fa47287b25a451" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e016b1c4cd81472c8432f8f34766874f"
I20260928 12:21:50.222278 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccc929ef361e48e58f25eaf639d5b921" candidate_uuid: "239e6f2672f64a35b0fa47287b25a451" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd35f7e208354f2f88192ad4c475b766"
I20260928 12:21:50.222358 9950 raft_consensus.cc:3037] T ccc929ef361e48e58f25eaf639d5b921 P e016b1c4cd81472c8432f8f34766874f [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:50.222358 10086 raft_consensus.cc:3037] T ccc929ef361e48e58f25eaf639d5b921 P cd35f7e208354f2f88192ad4c475b766 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:50.223228 9950 raft_consensus.cc:2454] T ccc929ef361e48e58f25eaf639d5b921 P e016b1c4cd81472c8432f8f34766874f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 239e6f2672f64a35b0fa47287b25a451 in term 1.
I20260928 12:21:50.223229 10086 raft_consensus.cc:2454] T ccc929ef361e48e58f25eaf639d5b921 P cd35f7e208354f2f88192ad4c475b766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 239e6f2672f64a35b0fa47287b25a451 in term 1.
I20260928 12:21:50.223404 10162 leader_election.cc:302] T ccc929ef361e48e58f25eaf639d5b921 P 239e6f2672f64a35b0fa47287b25a451 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 239e6f2672f64a35b0fa47287b25a451, e016b1c4cd81472c8432f8f34766874f; no voters:
I20260928 12:21:50.223500 10314 raft_consensus.cc:2781] T ccc929ef361e48e58f25eaf639d5b921 P 239e6f2672f64a35b0fa47287b25a451 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:21:50.223546 10314 raft_consensus.cc:686] T ccc929ef361e48e58f25eaf639d5b921 P 239e6f2672f64a35b0fa47287b25a451 [term 1 LEADER]: Becoming Leader. State: Replica: 239e6f2672f64a35b0fa47287b25a451, State: Running, Role: LEADER
I20260928 12:21:50.223598 10314 consensus_queue.cc:237] T ccc929ef361e48e58f25eaf639d5b921 P 239e6f2672f64a35b0fa47287b25a451 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } }
I20260928 12:21:50.224228 9793 catalog_manager.cc:5495] T ccc929ef361e48e58f25eaf639d5b921 P 239e6f2672f64a35b0fa47287b25a451 reported cstate change: term changed from 0 to 1, leader changed from <none> to 239e6f2672f64a35b0fa47287b25a451 (127.2.56.3). New cstate: current_term: 1 leader_uuid: "239e6f2672f64a35b0fa47287b25a451" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } health_report { overall_health: HEALTHY } } }
Sep 28 12:21:50 dist-test-slave-920p krb5kdc[9748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598109, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.62@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM
I20260928 12:21:50.285530 9861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:50.270071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35094 (local address 127.2.56.62:42227)
0928 12:21:50.270129 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:50.270131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:50.270238 (+ 107us) server_negotiation.cc:407] Connection header received
0928 12:21:50.270301 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:50.270301 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:50.270313 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:50.270360 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:50.270891 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:50.271433 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:50.272111 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:50.272291 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:50.272346 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:50.283111 (+ 10765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:50.283113 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:50.283113 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:50.283120 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:50.284334 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:50.284756 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:50.284757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:50.284757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:50.284808 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:50.284970 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:50.284970 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:50.284971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:50.285087 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:50.285144 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:50.285305 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 12:21:50.285392 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 12:21:50.285588 10339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:50.270012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:42227 (local address 127.0.0.1:35094)
0928 12:21:50.270164 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:50.270174 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:50.270247 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:50.270368 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:50.270369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:50.270754 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:50.270811 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:50.270814 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:50.271503 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:50.271504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:50.272041 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:50.272042 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:50.272346 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:50.272346 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:50.272395 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:50.281516 (+ 9121us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:50.281521 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:50.283029 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:50.284407 (+ 1378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:50.284415 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:50.284418 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:50.284686 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:50.284866 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:50.284867 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:50.284867 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:50.284924 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:50.285137 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:50.285144 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:50.285194 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 12:21:50.285253 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 12:21:50.285457 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":74,"thread_start_us":69,"threads_started":1}
I20260928 12:21:50.287861 9793 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:35094:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 12:21:50.288081 9793 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:21:50.289494 10210 tablet_service.cc:1467] Processing CreateTablet for tablet 220e524c743443c4b6ea3ee42681f120 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b4ba7c12d2cd4c9c85146b730cb66658]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 12:21:50.289479 9930 tablet_service.cc:1467] Processing CreateTablet for tablet 220e524c743443c4b6ea3ee42681f120 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b4ba7c12d2cd4c9c85146b730cb66658]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 12:21:50.289508 10070 tablet_service.cc:1467] Processing CreateTablet for tablet 220e524c743443c4b6ea3ee42681f120 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b4ba7c12d2cd4c9c85146b730cb66658]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 12:21:50.289662 9930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 220e524c743443c4b6ea3ee42681f120. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:50.289664 10210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 220e524c743443c4b6ea3ee42681f120. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:50.289664 10070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 220e524c743443c4b6ea3ee42681f120. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:50.290927 10305 tablet_bootstrap.cc:492] T 220e524c743443c4b6ea3ee42681f120 P cd35f7e208354f2f88192ad4c475b766: Bootstrap starting.
I20260928 12:21:50.291278 10304 tablet_bootstrap.cc:492] T 220e524c743443c4b6ea3ee42681f120 P 239e6f2672f64a35b0fa47287b25a451: Bootstrap starting.
I20260928 12:21:50.291347 10305 tablet_bootstrap.cc:654] T 220e524c743443c4b6ea3ee42681f120 P cd35f7e208354f2f88192ad4c475b766: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:50.291635 10304 tablet_bootstrap.cc:654] T 220e524c743443c4b6ea3ee42681f120 P 239e6f2672f64a35b0fa47287b25a451: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:50.291882 10306 tablet_bootstrap.cc:492] T 220e524c743443c4b6ea3ee42681f120 P e016b1c4cd81472c8432f8f34766874f: Bootstrap starting.
I20260928 12:21:50.292040 10305 tablet_bootstrap.cc:492] T 220e524c743443c4b6ea3ee42681f120 P cd35f7e208354f2f88192ad4c475b766: No bootstrap required, opened a new log
I20260928 12:21:50.292093 10305 ts_tablet_manager.cc:1397] T 220e524c743443c4b6ea3ee42681f120 P cd35f7e208354f2f88192ad4c475b766: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:21:50.292212 10304 tablet_bootstrap.cc:492] T 220e524c743443c4b6ea3ee42681f120 P 239e6f2672f64a35b0fa47287b25a451: No bootstrap required, opened a new log
I20260928 12:21:50.292263 10304 ts_tablet_manager.cc:1397] T 220e524c743443c4b6ea3ee42681f120 P 239e6f2672f64a35b0fa47287b25a451: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 12:21:50.292289 10306 tablet_bootstrap.cc:654] T 220e524c743443c4b6ea3ee42681f120 P e016b1c4cd81472c8432f8f34766874f: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:50.292269 10305 raft_consensus.cc:348] T 220e524c743443c4b6ea3ee42681f120 P cd35f7e208354f2f88192ad4c475b766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } }
I20260928 12:21:50.292346 10305 raft_consensus.cc:374] T 220e524c743443c4b6ea3ee42681f120 P cd35f7e208354f2f88192ad4c475b766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:50.292364 10305 raft_consensus.cc:729] T 220e524c743443c4b6ea3ee42681f120 P cd35f7e208354f2f88192ad4c475b766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd35f7e208354f2f88192ad4c475b766, State: Initialized, Role: FOLLOWER
I20260928 12:21:50.292376 10304 raft_consensus.cc:348] T 220e524c743443c4b6ea3ee42681f120 P 239e6f2672f64a35b0fa47287b25a451 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } }
I20260928 12:21:50.292433 10304 raft_consensus.cc:374] T 220e524c743443c4b6ea3ee42681f120 P 239e6f2672f64a35b0fa47287b25a451 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:50.292412 10305 consensus_queue.cc:260] T 220e524c743443c4b6ea3ee42681f120 P cd35f7e208354f2f88192ad4c475b766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } }
I20260928 12:21:50.292451 10304 raft_consensus.cc:729] T 220e524c743443c4b6ea3ee42681f120 P 239e6f2672f64a35b0fa47287b25a451 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 239e6f2672f64a35b0fa47287b25a451, State: Initialized, Role: FOLLOWER
I20260928 12:21:50.292490 10304 consensus_queue.cc:260] T 220e524c743443c4b6ea3ee42681f120 P 239e6f2672f64a35b0fa47287b25a451 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } }
I20260928 12:21:50.292523 10305 ts_tablet_manager.cc:1428] T 220e524c743443c4b6ea3ee42681f120 P cd35f7e208354f2f88192ad4c475b766: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:50.292579 10304 ts_tablet_manager.cc:1428] T 220e524c743443c4b6ea3ee42681f120 P 239e6f2672f64a35b0fa47287b25a451: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:50.292558 10311 tablet_replica.cc:439] T 220e524c743443c4b6ea3ee42681f120 P cd35f7e208354f2f88192ad4c475b766: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } } }
I20260928 12:21:50.292663 10311 tablet_replica.cc:442] T 220e524c743443c4b6ea3ee42681f120 P cd35f7e208354f2f88192ad4c475b766: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:21:50.292647 10314 tablet_replica.cc:439] T 220e524c743443c4b6ea3ee42681f120 P 239e6f2672f64a35b0fa47287b25a451: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } } }
I20260928 12:21:50.292716 10314 tablet_replica.cc:442] T 220e524c743443c4b6ea3ee42681f120 P 239e6f2672f64a35b0fa47287b25a451: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:21:50.293040 10306 tablet_bootstrap.cc:492] T 220e524c743443c4b6ea3ee42681f120 P e016b1c4cd81472c8432f8f34766874f: No bootstrap required, opened a new log
I20260928 12:21:50.293075 10306 ts_tablet_manager.cc:1397] T 220e524c743443c4b6ea3ee42681f120 P e016b1c4cd81472c8432f8f34766874f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 12:21:50.293231 10306 raft_consensus.cc:348] T 220e524c743443c4b6ea3ee42681f120 P e016b1c4cd81472c8432f8f34766874f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } }
I20260928 12:21:50.293320 10306 raft_consensus.cc:374] T 220e524c743443c4b6ea3ee42681f120 P e016b1c4cd81472c8432f8f34766874f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:50.293346 10306 raft_consensus.cc:729] T 220e524c743443c4b6ea3ee42681f120 P e016b1c4cd81472c8432f8f34766874f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e016b1c4cd81472c8432f8f34766874f, State: Initialized, Role: FOLLOWER
I20260928 12:21:50.293390 10306 consensus_queue.cc:260] T 220e524c743443c4b6ea3ee42681f120 P e016b1c4cd81472c8432f8f34766874f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } }
I20260928 12:21:50.293493 10306 ts_tablet_manager.cc:1428] T 220e524c743443c4b6ea3ee42681f120 P e016b1c4cd81472c8432f8f34766874f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:50.293555 10312 tablet_replica.cc:439] T 220e524c743443c4b6ea3ee42681f120 P e016b1c4cd81472c8432f8f34766874f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } } }
I20260928 12:21:50.293644 10312 tablet_replica.cc:442] T 220e524c743443c4b6ea3ee42681f120 P e016b1c4cd81472c8432f8f34766874f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:21:50.393648 10312 raft_consensus.cc:482] T 220e524c743443c4b6ea3ee42681f120 P e016b1c4cd81472c8432f8f34766874f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:21:50.393774 10312 raft_consensus.cc:504] T 220e524c743443c4b6ea3ee42681f120 P e016b1c4cd81472c8432f8f34766874f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } }
I20260928 12:21:50.394165 10312 leader_election.cc:288] T 220e524c743443c4b6ea3ee42681f120 P e016b1c4cd81472c8432f8f34766874f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd35f7e208354f2f88192ad4c475b766 (127.2.56.2:35533), 239e6f2672f64a35b0fa47287b25a451 (127.2.56.3:44701)
I20260928 12:21:50.397279 10301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:50.394576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:45075 (local address 127.2.56.2:35533)
0928 12:21:50.394623 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:50.394625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:50.394635 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:21:50.394714 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:50.394716 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:50.394735 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:50.394785 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:50.394974 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:50.395518 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:50.396573 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:50.396948 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:50.397007 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:50.397018 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:50.397145 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 12:21:50.397186 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 12:21:50.397284 10302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:50.394505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:52901 (local address 127.2.56.3:44701)
0928 12:21:50.394563 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:50.394565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:50.394691 (+ 126us) server_negotiation.cc:407] Connection header received
0928 12:21:50.394812 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:50.394813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:50.394824 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:50.394876 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:21:50.395013 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:50.395519 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:50.396561 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:50.396947 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:50.397007 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:50.397019 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:50.397145 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 12:21:50.397183 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 12:21:50.397327 10344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:50.394381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:44701 (local address 127.2.56.1:52901)
0928 12:21:50.394619 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:50.394630 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:50.394727 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:50.394891 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:50.394891 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:50.394893 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:50.394951 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:50.394952 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:50.395600 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:50.395602 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:50.396481 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:50.396484 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:50.397021 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:50.397022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:50.397085 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:50.397091 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 12:21:50.397135 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 12:21:50.397183 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":37,"thread_start_us":112,"threads_started":1}
I20260928 12:21:50.397342 10343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:50.394381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:35533 (local address 127.2.56.1:45075)
0928 12:21:50.394568 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:50.394581 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:50.394645 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:50.394807 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:50.394808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:50.394810 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:21:50.394902 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:50.394907 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:50.395599 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:50.395600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:50.396488 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:50.396490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:50.397015 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:50.397016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:50.397083 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:50.397087 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:21:50.397131 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 12:21:50.397181 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":40,"thread_start_us":110,"threads_started":1}
I20260928 12:21:50.397600 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220e524c743443c4b6ea3ee42681f120" candidate_uuid: "e016b1c4cd81472c8432f8f34766874f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd35f7e208354f2f88192ad4c475b766" is_pre_election: true
I20260928 12:21:50.397560 10230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220e524c743443c4b6ea3ee42681f120" candidate_uuid: "e016b1c4cd81472c8432f8f34766874f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "239e6f2672f64a35b0fa47287b25a451" is_pre_election: true
I20260928 12:21:50.397715 10086 raft_consensus.cc:2454] T 220e524c743443c4b6ea3ee42681f120 P cd35f7e208354f2f88192ad4c475b766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e016b1c4cd81472c8432f8f34766874f in term 0.
I20260928 12:21:50.397747 10230 raft_consensus.cc:2454] T 220e524c743443c4b6ea3ee42681f120 P 239e6f2672f64a35b0fa47287b25a451 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e016b1c4cd81472c8432f8f34766874f in term 0.
I20260928 12:21:50.397915 9884 leader_election.cc:302] T 220e524c743443c4b6ea3ee42681f120 P e016b1c4cd81472c8432f8f34766874f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd35f7e208354f2f88192ad4c475b766, e016b1c4cd81472c8432f8f34766874f; no voters:
I20260928 12:21:50.398078 10312 raft_consensus.cc:2781] T 220e524c743443c4b6ea3ee42681f120 P e016b1c4cd81472c8432f8f34766874f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:21:50.398120 10312 raft_consensus.cc:482] T 220e524c743443c4b6ea3ee42681f120 P e016b1c4cd81472c8432f8f34766874f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:21:50.398134 10312 raft_consensus.cc:3037] T 220e524c743443c4b6ea3ee42681f120 P e016b1c4cd81472c8432f8f34766874f [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:50.398885 10312 raft_consensus.cc:504] T 220e524c743443c4b6ea3ee42681f120 P e016b1c4cd81472c8432f8f34766874f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } }
I20260928 12:21:50.399012 10312 leader_election.cc:288] T 220e524c743443c4b6ea3ee42681f120 P e016b1c4cd81472c8432f8f34766874f [CANDIDATE]: Term 1 election: Requested vote from peers cd35f7e208354f2f88192ad4c475b766 (127.2.56.2:35533), 239e6f2672f64a35b0fa47287b25a451 (127.2.56.3:44701)
I20260928 12:21:50.399153 10086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220e524c743443c4b6ea3ee42681f120" candidate_uuid: "e016b1c4cd81472c8432f8f34766874f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd35f7e208354f2f88192ad4c475b766"
I20260928 12:21:50.399178 10230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220e524c743443c4b6ea3ee42681f120" candidate_uuid: "e016b1c4cd81472c8432f8f34766874f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "239e6f2672f64a35b0fa47287b25a451"
I20260928 12:21:50.399240 10230 raft_consensus.cc:3037] T 220e524c743443c4b6ea3ee42681f120 P 239e6f2672f64a35b0fa47287b25a451 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:50.399237 10086 raft_consensus.cc:3037] T 220e524c743443c4b6ea3ee42681f120 P cd35f7e208354f2f88192ad4c475b766 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:50.399857 10086 raft_consensus.cc:2454] T 220e524c743443c4b6ea3ee42681f120 P cd35f7e208354f2f88192ad4c475b766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e016b1c4cd81472c8432f8f34766874f in term 1.
I20260928 12:21:50.400020 10230 raft_consensus.cc:2454] T 220e524c743443c4b6ea3ee42681f120 P 239e6f2672f64a35b0fa47287b25a451 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e016b1c4cd81472c8432f8f34766874f in term 1.
I20260928 12:21:50.400035 9884 leader_election.cc:302] T 220e524c743443c4b6ea3ee42681f120 P e016b1c4cd81472c8432f8f34766874f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd35f7e208354f2f88192ad4c475b766, e016b1c4cd81472c8432f8f34766874f; no voters:
I20260928 12:21:50.400108 10312 raft_consensus.cc:2781] T 220e524c743443c4b6ea3ee42681f120 P e016b1c4cd81472c8432f8f34766874f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:21:50.400230 10312 raft_consensus.cc:686] T 220e524c743443c4b6ea3ee42681f120 P e016b1c4cd81472c8432f8f34766874f [term 1 LEADER]: Becoming Leader. State: Replica: e016b1c4cd81472c8432f8f34766874f, State: Running, Role: LEADER
I20260928 12:21:50.400301 10312 consensus_queue.cc:237] T 220e524c743443c4b6ea3ee42681f120 P e016b1c4cd81472c8432f8f34766874f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } }
I20260928 12:21:50.400697 10345 tablet_replica.cc:439] T 220e524c743443c4b6ea3ee42681f120 P e016b1c4cd81472c8432f8f34766874f: TxnStatusTablet state changed. Reason: New leader e016b1c4cd81472c8432f8f34766874f. Latest consensus state: current_term: 1 leader_uuid: "e016b1c4cd81472c8432f8f34766874f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } } }
I20260928 12:21:50.400777 10345 tablet_replica.cc:442] T 220e524c743443c4b6ea3ee42681f120 P e016b1c4cd81472c8432f8f34766874f: This TxnStatusTablet replica's current role is: LEADER
I20260928 12:21:50.400962 10347 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 12:21:50.401077 10347 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 12:21:50.401113 9793 catalog_manager.cc:5495] T 220e524c743443c4b6ea3ee42681f120 P e016b1c4cd81472c8432f8f34766874f reported cstate change: term changed from 0 to 1, leader changed from <none> to e016b1c4cd81472c8432f8f34766874f (127.2.56.1). New cstate: current_term: 1 leader_uuid: "e016b1c4cd81472c8432f8f34766874f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } health_report { overall_health: UNKNOWN } } }
I20260928 12:21:50.408471 10339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:50.405686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:44449 (local address 127.0.0.1:58386)
0928 12:21:50.405781 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:50.405791 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:50.405828 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:50.405988 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:50.405988 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:50.405990 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:21:50.406042 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:50.406045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:50.406959 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:50.406960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:50.407544 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:50.407546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:50.407996 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:50.407997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:50.408047 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:50.408062 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:21:50.408335 (+ 273us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:21:50.408339 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:21:50.408362 (+ 23us) client_negotiation.cc:240] Negotiation successful
0928 12:21:50.408404 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 12:21:50.408481 10303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:50.405766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58386 (local address 127.2.56.1:44449)
0928 12:21:50.405819 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:50.405821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:50.405831 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:21:50.405889 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:50.405889 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:50.405908 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:50.405987 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:21:50.406083 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:50.406884 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:50.407621 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:50.407926 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:50.407998 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:50.408149 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:21:50.408279 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:21:50.408356 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:50.408375 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 12:21:50.408399 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 12:21:50.409624 10086 raft_consensus.cc:1261] T 220e524c743443c4b6ea3ee42681f120 P cd35f7e208354f2f88192ad4c475b766 [term 1 FOLLOWER]: Refusing update from remote peer e016b1c4cd81472c8432f8f34766874f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:21:50.409845 10230 raft_consensus.cc:1261] T 220e524c743443c4b6ea3ee42681f120 P 239e6f2672f64a35b0fa47287b25a451 [term 1 FOLLOWER]: Refusing update from remote peer e016b1c4cd81472c8432f8f34766874f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:21:50.409883 10345 consensus_queue.cc:1035] T 220e524c743443c4b6ea3ee42681f120 P e016b1c4cd81472c8432f8f34766874f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:21:50.410092 10345 consensus_queue.cc:1035] T 220e524c743443c4b6ea3ee42681f120 P e016b1c4cd81472c8432f8f34766874f [LEADER]: Connected to new peer: Peer: permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:21:50.410487 10311 tablet_replica.cc:439] T 220e524c743443c4b6ea3ee42681f120 P cd35f7e208354f2f88192ad4c475b766: TxnStatusTablet state changed. Reason: New leader e016b1c4cd81472c8432f8f34766874f. Latest consensus state: current_term: 1 leader_uuid: "e016b1c4cd81472c8432f8f34766874f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } } }
I20260928 12:21:50.410583 10311 tablet_replica.cc:442] T 220e524c743443c4b6ea3ee42681f120 P cd35f7e208354f2f88192ad4c475b766: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:21:50.410635 10314 tablet_replica.cc:439] T 220e524c743443c4b6ea3ee42681f120 P 239e6f2672f64a35b0fa47287b25a451: TxnStatusTablet state changed. Reason: New leader e016b1c4cd81472c8432f8f34766874f. Latest consensus state: current_term: 1 leader_uuid: "e016b1c4cd81472c8432f8f34766874f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } } }
I20260928 12:21:50.410758 10314 tablet_replica.cc:442] T 220e524c743443c4b6ea3ee42681f120 P 239e6f2672f64a35b0fa47287b25a451: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:21:50.411396 10312 tablet_replica.cc:439] T 220e524c743443c4b6ea3ee42681f120 P e016b1c4cd81472c8432f8f34766874f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e016b1c4cd81472c8432f8f34766874f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } } }
I20260928 12:21:50.411484 10312 tablet_replica.cc:442] T 220e524c743443c4b6ea3ee42681f120 P e016b1c4cd81472c8432f8f34766874f: This TxnStatusTablet replica's current role is: LEADER
I20260928 12:21:50.411458 10318 mvcc.cc:205] Tried to move back new op lower bound from 7334289860235579392 to 7334289860200169472. Current Snapshot: MvccSnapshot[applied={T|T < 7334289860235579392}]
I20260928 12:21:50.411991 10314 tablet_replica.cc:439] T 220e524c743443c4b6ea3ee42681f120 P 239e6f2672f64a35b0fa47287b25a451: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e016b1c4cd81472c8432f8f34766874f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } } }
I20260928 12:21:50.412077 10314 tablet_replica.cc:442] T 220e524c743443c4b6ea3ee42681f120 P 239e6f2672f64a35b0fa47287b25a451: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:21:50.413694 10345 tablet_replica.cc:439] T 220e524c743443c4b6ea3ee42681f120 P e016b1c4cd81472c8432f8f34766874f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e016b1c4cd81472c8432f8f34766874f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } } }
I20260928 12:21:50.413798 10345 tablet_replica.cc:442] T 220e524c743443c4b6ea3ee42681f120 P e016b1c4cd81472c8432f8f34766874f: This TxnStatusTablet replica's current role is: LEADER
I20260928 12:21:50.414232 10311 tablet_replica.cc:439] T 220e524c743443c4b6ea3ee42681f120 P cd35f7e208354f2f88192ad4c475b766: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e016b1c4cd81472c8432f8f34766874f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } } }
I20260928 12:21:50.414326 10311 tablet_replica.cc:442] T 220e524c743443c4b6ea3ee42681f120 P cd35f7e208354f2f88192ad4c475b766: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:21:50.418304 10303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:50.415659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:52623 (local address 127.2.56.1:44449)
0928 12:21:50.415704 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:50.415706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:50.415766 (+ 60us) server_negotiation.cc:407] Connection header received
0928 12:21:50.415817 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:50.415818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:50.415828 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:50.415884 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:21:50.415984 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:50.416749 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:50.417544 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:50.417849 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:50.417915 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:50.418081 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:21:50.418119 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:21:50.418162 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:50.418212 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 12:21:50.418234 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:21:50.418339 10282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:50.415596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:44449 (local address 127.2.56.3:52623)
0928 12:21:50.415723 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:50.415733 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:50.415776 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:50.415886 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:50.415886 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:50.415888 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:21:50.415949 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:50.415950 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:50.416816 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:50.416816 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:50.417480 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:50.417483 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:50.417921 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:50.417922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:50.417966 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:50.417982 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:21:50.418168 (+ 186us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:21:50.418170 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:21:50.418210 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 12:21:50.418273 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260928 12:21:50.420552 10086 raft_consensus.cc:1261] T ccc929ef361e48e58f25eaf639d5b921 P cd35f7e208354f2f88192ad4c475b766 [term 1 FOLLOWER]: Refusing update from remote peer 239e6f2672f64a35b0fa47287b25a451: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:21:50.420567 9950 raft_consensus.cc:1261] T ccc929ef361e48e58f25eaf639d5b921 P e016b1c4cd81472c8432f8f34766874f [term 1 FOLLOWER]: Refusing update from remote peer 239e6f2672f64a35b0fa47287b25a451: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:21:50.420727 10310 consensus_queue.cc:1035] T ccc929ef361e48e58f25eaf639d5b921 P 239e6f2672f64a35b0fa47287b25a451 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:21:50.420836 10310 consensus_queue.cc:1035] T ccc929ef361e48e58f25eaf639d5b921 P 239e6f2672f64a35b0fa47287b25a451 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:21:50.421898 10317 mvcc.cc:205] Tried to move back new op lower bound from 7334289860281397248 to 7334289859476176896. Current Snapshot: MvccSnapshot[applied={T|T < 7334289860281397248}]
I20260928 12:21:50.429195 10302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:50.426400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:60381 (local address 127.2.56.3:44701)
0928 12:21:50.426449 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:50.426463 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:50.426665 (+ 202us) server_negotiation.cc:407] Connection header received
0928 12:21:50.426725 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:50.426725 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:50.426737 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:50.426780 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:21:50.426948 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:50.427424 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:50.428595 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:50.428786 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:50.428836 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:50.429003 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:21:50.429031 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:21:50.429061 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:50.429113 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 12:21:50.429133 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:21:50.429517 10358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:50.426380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:44701 (local address 127.2.56.1:60381)
0928 12:21:50.426590 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:50.426603 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:50.426670 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:50.426782 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:50.426783 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:50.426795 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:21:50.426893 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:50.426895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:50.427486 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:50.427487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:50.428523 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:50.428526 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:50.428847 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:50.428848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:50.428915 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:50.428933 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:21:50.429064 (+ 131us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:21:50.429065 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 12:21:50.429110 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 12:21:50.429156 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":124,"threads_started":1}
I20260928 12:21:50.437986 9793 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:21:50.438102 9793 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:21:50.438755 9793 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 049b783e402e4560b7d3b9b60de4b55b: Sending DeleteTablet for 3 replicas of tablet ccc929ef361e48e58f25eaf639d5b921
I20260928 12:21:50.438979 9930 tablet_service.cc:1514] Processing DeleteTablet for tablet ccc929ef361e48e58f25eaf639d5b921 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:50 UTC) from {username='oryx', principal='oryx/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:58382
I20260928 12:21:50.439008 10070 tablet_service.cc:1514] Processing DeleteTablet for tablet ccc929ef361e48e58f25eaf639d5b921 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:50 UTC) from {username='oryx', principal='oryx/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:56496
I20260928 12:21:50.439080 10327 tablet_replica.cc:330] T ccc929ef361e48e58f25eaf639d5b921 P e016b1c4cd81472c8432f8f34766874f: stopping tablet replica
I20260928 12:21:50.439114 10329 tablet_replica.cc:330] T ccc929ef361e48e58f25eaf639d5b921 P cd35f7e208354f2f88192ad4c475b766: stopping tablet replica
I20260928 12:21:50.439139 10327 raft_consensus.cc:2229] T ccc929ef361e48e58f25eaf639d5b921 P e016b1c4cd81472c8432f8f34766874f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:21:50.439172 10327 raft_consensus.cc:2258] T ccc929ef361e48e58f25eaf639d5b921 P e016b1c4cd81472c8432f8f34766874f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:50.439167 10329 raft_consensus.cc:2229] T ccc929ef361e48e58f25eaf639d5b921 P cd35f7e208354f2f88192ad4c475b766 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:21:50.439153 10210 tablet_service.cc:1514] Processing DeleteTablet for tablet ccc929ef361e48e58f25eaf639d5b921 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:50 UTC) from {username='oryx', principal='oryx/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:42046
I20260928 12:21:50.439203 10329 raft_consensus.cc:2258] T ccc929ef361e48e58f25eaf639d5b921 P cd35f7e208354f2f88192ad4c475b766 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:50.439239 10328 tablet_replica.cc:330] T ccc929ef361e48e58f25eaf639d5b921 P 239e6f2672f64a35b0fa47287b25a451: stopping tablet replica
I20260928 12:21:50.439293 10328 raft_consensus.cc:2229] T ccc929ef361e48e58f25eaf639d5b921 P 239e6f2672f64a35b0fa47287b25a451 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:21:50.439347 10328 raft_consensus.cc:2258] T ccc929ef361e48e58f25eaf639d5b921 P 239e6f2672f64a35b0fa47287b25a451 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:50.439494 10329 ts_tablet_manager.cc:1905] T ccc929ef361e48e58f25eaf639d5b921 P cd35f7e208354f2f88192ad4c475b766: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:50.439517 10327 ts_tablet_manager.cc:1905] T ccc929ef361e48e58f25eaf639d5b921 P e016b1c4cd81472c8432f8f34766874f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:50.439845 10328 ts_tablet_manager.cc:1905] T ccc929ef361e48e58f25eaf639d5b921 P 239e6f2672f64a35b0fa47287b25a451: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:50.441423 10329 ts_tablet_manager.cc:1918] T ccc929ef361e48e58f25eaf639d5b921 P cd35f7e208354f2f88192ad4c475b766: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:21:50.441430 10327 ts_tablet_manager.cc:1918] T ccc929ef361e48e58f25eaf639d5b921 P e016b1c4cd81472c8432f8f34766874f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:21:50.441476 10329 log.cc:1196] T ccc929ef361e48e58f25eaf639d5b921 P cd35f7e208354f2f88192ad4c475b766: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-1/wal/wals/ccc929ef361e48e58f25eaf639d5b921
I20260928 12:21:50.441476 10327 log.cc:1196] T ccc929ef361e48e58f25eaf639d5b921 P e016b1c4cd81472c8432f8f34766874f: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-0/wal/wals/ccc929ef361e48e58f25eaf639d5b921
I20260928 12:21:50.441620 10328 ts_tablet_manager.cc:1918] T ccc929ef361e48e58f25eaf639d5b921 P 239e6f2672f64a35b0fa47287b25a451: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:21:50.441669 10328 log.cc:1196] T ccc929ef361e48e58f25eaf639d5b921 P 239e6f2672f64a35b0fa47287b25a451: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-2/wal/wals/ccc929ef361e48e58f25eaf639d5b921
I20260928 12:21:50.441684 10329 ts_tablet_manager.cc:1939] T ccc929ef361e48e58f25eaf639d5b921 P cd35f7e208354f2f88192ad4c475b766: Deleting consensus metadata
I20260928 12:21:50.441685 10327 ts_tablet_manager.cc:1939] T ccc929ef361e48e58f25eaf639d5b921 P e016b1c4cd81472c8432f8f34766874f: Deleting consensus metadata
I20260928 12:21:50.441875 10328 ts_tablet_manager.cc:1939] T ccc929ef361e48e58f25eaf639d5b921 P 239e6f2672f64a35b0fa47287b25a451: Deleting consensus metadata
I20260928 12:21:50.442283 9779 catalog_manager.cc:4841] TS 239e6f2672f64a35b0fa47287b25a451 (127.2.56.3:44701): tablet ccc929ef361e48e58f25eaf639d5b921 (table test-table [id=1c8afd4143b54988aa627895408f3faa]) successfully deleted
I20260928 12:21:50.442489 9778 catalog_manager.cc:4841] TS e016b1c4cd81472c8432f8f34766874f (127.2.56.1:44449): tablet ccc929ef361e48e58f25eaf639d5b921 (table test-table [id=1c8afd4143b54988aa627895408f3faa]) successfully deleted
I20260928 12:21:50.444600 9780 catalog_manager.cc:4841] TS cd35f7e208354f2f88192ad4c475b766 (127.2.56.2:35533): tablet ccc929ef361e48e58f25eaf639d5b921 (table test-table [id=1c8afd4143b54988aa627895408f3faa]) successfully deleted
Sep 28 12:21:50 dist-test-slave-920p krb5kdc[9748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 12:21:50 dist-test-slave-920p krb5kdc[9748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:21:50 dist-test-slave-920p krb5kdc[9748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 12:21:50.465438 9861 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:21:50.459814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35096 (local address 127.2.56.62:42227)
0928 12:21:50.459873 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:50.459875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:50.460009 (+ 134us) server_negotiation.cc:407] Connection header received
0928 12:21:50.460089 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:50.460090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:50.460104 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:50.460152 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:50.460733 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:50.461313 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:50.462098 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:50.462304 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:50.462358 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:50.465326 (+ 2968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35096: BlockingRecv error: recv got EOF from 127.0.0.1:35096 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 28 12:21:50 dist-test-slave-920p krb5kdc[9748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM
I20260928 12:21:50.473309 9861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:50.466183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35102 (local address 127.2.56.62:42227)
0928 12:21:50.466222 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:50.466224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:50.466323 (+ 99us) server_negotiation.cc:407] Connection header received
0928 12:21:50.466391 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:50.466391 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:50.466400 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:50.466445 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:50.467029 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:50.467501 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:50.468482 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:50.468654 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:50.468695 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:50.470978 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:50.470980 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:50.470981 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:50.470988 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:50.472232 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:50.472532 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:50.472533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:50.472534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:50.472584 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:50.472742 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:50.472743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:50.472743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:50.472899 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:50.472999 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:50.473101 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 12:21:50.473179 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:21:50.474836 9793 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35102:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:21:50.475015 9793 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:21:50.476526 9930 tablet_service.cc:1467] Processing CreateTablet for tablet e1478292dd0340dc8030536ae3d23211 (DEFAULT_TABLE table=test-table [id=4ee297fa1b294525b98ad7d88a699246]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:50.476589 10210 tablet_service.cc:1467] Processing CreateTablet for tablet e1478292dd0340dc8030536ae3d23211 (DEFAULT_TABLE table=test-table [id=4ee297fa1b294525b98ad7d88a699246]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:50.476658 9930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1478292dd0340dc8030536ae3d23211. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:50.476751 10210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1478292dd0340dc8030536ae3d23211. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:50.478055 10306 tablet_bootstrap.cc:492] T e1478292dd0340dc8030536ae3d23211 P e016b1c4cd81472c8432f8f34766874f: Bootstrap starting.
I20260928 12:21:50.478314 10304 tablet_bootstrap.cc:492] T e1478292dd0340dc8030536ae3d23211 P 239e6f2672f64a35b0fa47287b25a451: Bootstrap starting.
I20260928 12:21:50.478415 10306 tablet_bootstrap.cc:654] T e1478292dd0340dc8030536ae3d23211 P e016b1c4cd81472c8432f8f34766874f: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:50.478415 10070 tablet_service.cc:1467] Processing CreateTablet for tablet e1478292dd0340dc8030536ae3d23211 (DEFAULT_TABLE table=test-table [id=4ee297fa1b294525b98ad7d88a699246]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:50.478549 10070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1478292dd0340dc8030536ae3d23211. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:50.478642 10304 tablet_bootstrap.cc:654] T e1478292dd0340dc8030536ae3d23211 P 239e6f2672f64a35b0fa47287b25a451: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:50.479157 10304 tablet_bootstrap.cc:492] T e1478292dd0340dc8030536ae3d23211 P 239e6f2672f64a35b0fa47287b25a451: No bootstrap required, opened a new log
I20260928 12:21:50.479189 10304 ts_tablet_manager.cc:1397] T e1478292dd0340dc8030536ae3d23211 P 239e6f2672f64a35b0fa47287b25a451: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 12:21:50.479286 10304 raft_consensus.cc:348] T e1478292dd0340dc8030536ae3d23211 P 239e6f2672f64a35b0fa47287b25a451 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } }
I20260928 12:21:50.479337 10304 raft_consensus.cc:374] T e1478292dd0340dc8030536ae3d23211 P 239e6f2672f64a35b0fa47287b25a451 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:50.479354 10304 raft_consensus.cc:729] T e1478292dd0340dc8030536ae3d23211 P 239e6f2672f64a35b0fa47287b25a451 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 239e6f2672f64a35b0fa47287b25a451, State: Initialized, Role: FOLLOWER
I20260928 12:21:50.479384 10304 consensus_queue.cc:260] T e1478292dd0340dc8030536ae3d23211 P 239e6f2672f64a35b0fa47287b25a451 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } }
I20260928 12:21:50.479480 10304 ts_tablet_manager.cc:1428] T e1478292dd0340dc8030536ae3d23211 P 239e6f2672f64a35b0fa47287b25a451: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:50.479975 10305 tablet_bootstrap.cc:492] T e1478292dd0340dc8030536ae3d23211 P cd35f7e208354f2f88192ad4c475b766: Bootstrap starting.
I20260928 12:21:50.480346 10305 tablet_bootstrap.cc:654] T e1478292dd0340dc8030536ae3d23211 P cd35f7e208354f2f88192ad4c475b766: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:50.480649 10306 tablet_bootstrap.cc:492] T e1478292dd0340dc8030536ae3d23211 P e016b1c4cd81472c8432f8f34766874f: No bootstrap required, opened a new log
I20260928 12:21:50.480701 10306 ts_tablet_manager.cc:1397] T e1478292dd0340dc8030536ae3d23211 P e016b1c4cd81472c8432f8f34766874f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 12:21:50.480854 10306 raft_consensus.cc:348] T e1478292dd0340dc8030536ae3d23211 P e016b1c4cd81472c8432f8f34766874f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } }
I20260928 12:21:50.480921 10306 raft_consensus.cc:374] T e1478292dd0340dc8030536ae3d23211 P e016b1c4cd81472c8432f8f34766874f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:50.480945 10306 raft_consensus.cc:729] T e1478292dd0340dc8030536ae3d23211 P e016b1c4cd81472c8432f8f34766874f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e016b1c4cd81472c8432f8f34766874f, State: Initialized, Role: FOLLOWER
I20260928 12:21:50.480988 10306 consensus_queue.cc:260] T e1478292dd0340dc8030536ae3d23211 P e016b1c4cd81472c8432f8f34766874f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } }
I20260928 12:21:50.481067 10306 ts_tablet_manager.cc:1428] T e1478292dd0340dc8030536ae3d23211 P e016b1c4cd81472c8432f8f34766874f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:50.481647 10305 tablet_bootstrap.cc:492] T e1478292dd0340dc8030536ae3d23211 P cd35f7e208354f2f88192ad4c475b766: No bootstrap required, opened a new log
I20260928 12:21:50.481694 10305 ts_tablet_manager.cc:1397] T e1478292dd0340dc8030536ae3d23211 P cd35f7e208354f2f88192ad4c475b766: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:21:50.481860 10305 raft_consensus.cc:348] T e1478292dd0340dc8030536ae3d23211 P cd35f7e208354f2f88192ad4c475b766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } }
I20260928 12:21:50.481940 10305 raft_consensus.cc:374] T e1478292dd0340dc8030536ae3d23211 P cd35f7e208354f2f88192ad4c475b766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:50.481954 10305 raft_consensus.cc:729] T e1478292dd0340dc8030536ae3d23211 P cd35f7e208354f2f88192ad4c475b766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd35f7e208354f2f88192ad4c475b766, State: Initialized, Role: FOLLOWER
I20260928 12:21:50.481989 10305 consensus_queue.cc:260] T e1478292dd0340dc8030536ae3d23211 P cd35f7e208354f2f88192ad4c475b766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } }
I20260928 12:21:50.482076 10305 ts_tablet_manager.cc:1428] T e1478292dd0340dc8030536ae3d23211 P cd35f7e208354f2f88192ad4c475b766: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:50.587610 10322 raft_consensus.cc:482] T e1478292dd0340dc8030536ae3d23211 P 239e6f2672f64a35b0fa47287b25a451 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:21:50.587720 10322 raft_consensus.cc:504] T e1478292dd0340dc8030536ae3d23211 P 239e6f2672f64a35b0fa47287b25a451 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } }
I20260928 12:21:50.587884 10322 leader_election.cc:288] T e1478292dd0340dc8030536ae3d23211 P 239e6f2672f64a35b0fa47287b25a451 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e016b1c4cd81472c8432f8f34766874f (127.2.56.1:44449), cd35f7e208354f2f88192ad4c475b766 (127.2.56.2:35533)
I20260928 12:21:50.588142 9950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1478292dd0340dc8030536ae3d23211" candidate_uuid: "239e6f2672f64a35b0fa47287b25a451" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e016b1c4cd81472c8432f8f34766874f" is_pre_election: true
I20260928 12:21:50.588254 9950 raft_consensus.cc:2454] T e1478292dd0340dc8030536ae3d23211 P e016b1c4cd81472c8432f8f34766874f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 239e6f2672f64a35b0fa47287b25a451 in term 0.
I20260928 12:21:50.588187 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1478292dd0340dc8030536ae3d23211" candidate_uuid: "239e6f2672f64a35b0fa47287b25a451" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd35f7e208354f2f88192ad4c475b766" is_pre_election: true
I20260928 12:21:50.588315 10088 raft_consensus.cc:2454] T e1478292dd0340dc8030536ae3d23211 P cd35f7e208354f2f88192ad4c475b766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 239e6f2672f64a35b0fa47287b25a451 in term 0.
I20260928 12:21:50.588457 10162 leader_election.cc:302] T e1478292dd0340dc8030536ae3d23211 P 239e6f2672f64a35b0fa47287b25a451 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 239e6f2672f64a35b0fa47287b25a451, e016b1c4cd81472c8432f8f34766874f; no voters:
I20260928 12:21:50.588550 10322 raft_consensus.cc:2781] T e1478292dd0340dc8030536ae3d23211 P 239e6f2672f64a35b0fa47287b25a451 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:21:50.588593 10322 raft_consensus.cc:482] T e1478292dd0340dc8030536ae3d23211 P 239e6f2672f64a35b0fa47287b25a451 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:21:50.588608 10322 raft_consensus.cc:3037] T e1478292dd0340dc8030536ae3d23211 P 239e6f2672f64a35b0fa47287b25a451 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:50.589507 10322 raft_consensus.cc:504] T e1478292dd0340dc8030536ae3d23211 P 239e6f2672f64a35b0fa47287b25a451 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } }
I20260928 12:21:50.589644 10322 leader_election.cc:288] T e1478292dd0340dc8030536ae3d23211 P 239e6f2672f64a35b0fa47287b25a451 [CANDIDATE]: Term 1 election: Requested vote from peers e016b1c4cd81472c8432f8f34766874f (127.2.56.1:44449), cd35f7e208354f2f88192ad4c475b766 (127.2.56.2:35533)
I20260928 12:21:50.589788 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1478292dd0340dc8030536ae3d23211" candidate_uuid: "239e6f2672f64a35b0fa47287b25a451" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd35f7e208354f2f88192ad4c475b766"
I20260928 12:21:50.589787 9950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1478292dd0340dc8030536ae3d23211" candidate_uuid: "239e6f2672f64a35b0fa47287b25a451" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e016b1c4cd81472c8432f8f34766874f"
I20260928 12:21:50.589854 9950 raft_consensus.cc:3037] T e1478292dd0340dc8030536ae3d23211 P e016b1c4cd81472c8432f8f34766874f [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:50.589851 10088 raft_consensus.cc:3037] T e1478292dd0340dc8030536ae3d23211 P cd35f7e208354f2f88192ad4c475b766 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:50.590523 9950 raft_consensus.cc:2454] T e1478292dd0340dc8030536ae3d23211 P e016b1c4cd81472c8432f8f34766874f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 239e6f2672f64a35b0fa47287b25a451 in term 1.
I20260928 12:21:50.590585 10088 raft_consensus.cc:2454] T e1478292dd0340dc8030536ae3d23211 P cd35f7e208354f2f88192ad4c475b766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 239e6f2672f64a35b0fa47287b25a451 in term 1.
I20260928 12:21:50.590713 10162 leader_election.cc:302] T e1478292dd0340dc8030536ae3d23211 P 239e6f2672f64a35b0fa47287b25a451 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 239e6f2672f64a35b0fa47287b25a451, e016b1c4cd81472c8432f8f34766874f; no voters:
I20260928 12:21:50.590821 10322 raft_consensus.cc:2781] T e1478292dd0340dc8030536ae3d23211 P 239e6f2672f64a35b0fa47287b25a451 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:21:50.590890 10322 raft_consensus.cc:686] T e1478292dd0340dc8030536ae3d23211 P 239e6f2672f64a35b0fa47287b25a451 [term 1 LEADER]: Becoming Leader. State: Replica: 239e6f2672f64a35b0fa47287b25a451, State: Running, Role: LEADER
I20260928 12:21:50.590951 10322 consensus_queue.cc:237] T e1478292dd0340dc8030536ae3d23211 P 239e6f2672f64a35b0fa47287b25a451 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } }
I20260928 12:21:50.591480 9791 catalog_manager.cc:5495] T e1478292dd0340dc8030536ae3d23211 P 239e6f2672f64a35b0fa47287b25a451 reported cstate change: term changed from 0 to 1, leader changed from <none> to 239e6f2672f64a35b0fa47287b25a451 (127.2.56.3). New cstate: current_term: 1 leader_uuid: "239e6f2672f64a35b0fa47287b25a451" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } health_report { overall_health: HEALTHY } } }
I20260928 12:21:50.624251 10302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:50.622186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42068 (local address 127.2.56.3:44701)
0928 12:21:50.622243 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:50.622245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:50.622252 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:21:50.622278 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:50.622279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:50.622293 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:50.622340 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:21:50.622462 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:50.622970 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:50.623686 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:50.623873 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:50.623931 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:50.624044 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:21:50.624073 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:21:50.624105 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:50.624160 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 12:21:50.624178 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 12:21:50.625167 9950 raft_consensus.cc:1261] T e1478292dd0340dc8030536ae3d23211 P e016b1c4cd81472c8432f8f34766874f [term 1 FOLLOWER]: Refusing update from remote peer 239e6f2672f64a35b0fa47287b25a451: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:21:50.625167 10088 raft_consensus.cc:1261] T e1478292dd0340dc8030536ae3d23211 P cd35f7e208354f2f88192ad4c475b766 [term 1 FOLLOWER]: Refusing update from remote peer 239e6f2672f64a35b0fa47287b25a451: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:21:50.625455 10322 consensus_queue.cc:1035] T e1478292dd0340dc8030536ae3d23211 P 239e6f2672f64a35b0fa47287b25a451 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:21:50.625577 10310 consensus_queue.cc:1035] T e1478292dd0340dc8030536ae3d23211 P 239e6f2672f64a35b0fa47287b25a451 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:21:50.627895 9793 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:21:50.627987 9793 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:21:50.628569 9793 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 049b783e402e4560b7d3b9b60de4b55b: Sending DeleteTablet for 3 replicas of tablet e1478292dd0340dc8030536ae3d23211
I20260928 12:21:50.628840 10070 tablet_service.cc:1514] Processing DeleteTablet for tablet e1478292dd0340dc8030536ae3d23211 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:50 UTC) from {username='oryx', principal='oryx/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:56496
I20260928 12:21:50.628849 10210 tablet_service.cc:1514] Processing DeleteTablet for tablet e1478292dd0340dc8030536ae3d23211 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:50 UTC) from {username='oryx', principal='oryx/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:42046
I20260928 12:21:50.628939 10329 tablet_replica.cc:330] T e1478292dd0340dc8030536ae3d23211 P cd35f7e208354f2f88192ad4c475b766: stopping tablet replica
I20260928 12:21:50.628939 10328 tablet_replica.cc:330] T e1478292dd0340dc8030536ae3d23211 P 239e6f2672f64a35b0fa47287b25a451: stopping tablet replica
I20260928 12:21:50.628988 10329 raft_consensus.cc:2229] T e1478292dd0340dc8030536ae3d23211 P cd35f7e208354f2f88192ad4c475b766 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:21:50.628993 10328 raft_consensus.cc:2229] T e1478292dd0340dc8030536ae3d23211 P 239e6f2672f64a35b0fa47287b25a451 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:21:50.629045 10329 raft_consensus.cc:2258] T e1478292dd0340dc8030536ae3d23211 P cd35f7e208354f2f88192ad4c475b766 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:50.629053 10328 raft_consensus.cc:2258] T e1478292dd0340dc8030536ae3d23211 P 239e6f2672f64a35b0fa47287b25a451 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:50.629040 9793 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35102:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 12:21:50.629129 9930 tablet_service.cc:1514] Processing DeleteTablet for tablet e1478292dd0340dc8030536ae3d23211 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:50 UTC) from {username='oryx', principal='oryx/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:58382
W20260928 12:21:50.629182 9793 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:21:50.629212 10327 tablet_replica.cc:330] T e1478292dd0340dc8030536ae3d23211 P e016b1c4cd81472c8432f8f34766874f: stopping tablet replica
I20260928 12:21:50.629294 10327 raft_consensus.cc:2229] T e1478292dd0340dc8030536ae3d23211 P e016b1c4cd81472c8432f8f34766874f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:21:50.629333 10327 raft_consensus.cc:2258] T e1478292dd0340dc8030536ae3d23211 P e016b1c4cd81472c8432f8f34766874f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:50.629583 10328 ts_tablet_manager.cc:1905] T e1478292dd0340dc8030536ae3d23211 P 239e6f2672f64a35b0fa47287b25a451: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:50.629714 10329 ts_tablet_manager.cc:1905] T e1478292dd0340dc8030536ae3d23211 P cd35f7e208354f2f88192ad4c475b766: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:50.629774 10327 ts_tablet_manager.cc:1905] T e1478292dd0340dc8030536ae3d23211 P e016b1c4cd81472c8432f8f34766874f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:50.630437 10070 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab8ee3bf68c46ea9c223c16b3cb30e5 (DEFAULT_TABLE table=test-table [id=b32e09eddc68403ca6d871bf14fd8678]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:50.630550 10070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab8ee3bf68c46ea9c223c16b3cb30e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:50.631279 10329 ts_tablet_manager.cc:1918] T e1478292dd0340dc8030536ae3d23211 P cd35f7e208354f2f88192ad4c475b766: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:21:50.631326 10329 log.cc:1196] T e1478292dd0340dc8030536ae3d23211 P cd35f7e208354f2f88192ad4c475b766: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-1/wal/wals/e1478292dd0340dc8030536ae3d23211
I20260928 12:21:50.631319 10327 ts_tablet_manager.cc:1918] T e1478292dd0340dc8030536ae3d23211 P e016b1c4cd81472c8432f8f34766874f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:21:50.631319 10328 ts_tablet_manager.cc:1918] T e1478292dd0340dc8030536ae3d23211 P 239e6f2672f64a35b0fa47287b25a451: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:21:50.631369 10327 log.cc:1196] T e1478292dd0340dc8030536ae3d23211 P e016b1c4cd81472c8432f8f34766874f: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-0/wal/wals/e1478292dd0340dc8030536ae3d23211
I20260928 12:21:50.631369 10328 log.cc:1196] T e1478292dd0340dc8030536ae3d23211 P 239e6f2672f64a35b0fa47287b25a451: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-2/wal/wals/e1478292dd0340dc8030536ae3d23211
I20260928 12:21:50.631541 10329 ts_tablet_manager.cc:1939] T e1478292dd0340dc8030536ae3d23211 P cd35f7e208354f2f88192ad4c475b766: Deleting consensus metadata
I20260928 12:21:50.631578 10327 ts_tablet_manager.cc:1939] T e1478292dd0340dc8030536ae3d23211 P e016b1c4cd81472c8432f8f34766874f: Deleting consensus metadata
I20260928 12:21:50.631578 10328 ts_tablet_manager.cc:1939] T e1478292dd0340dc8030536ae3d23211 P 239e6f2672f64a35b0fa47287b25a451: Deleting consensus metadata
I20260928 12:21:50.631935 10305 tablet_bootstrap.cc:492] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P cd35f7e208354f2f88192ad4c475b766: Bootstrap starting.
I20260928 12:21:50.632107 9780 catalog_manager.cc:4841] TS cd35f7e208354f2f88192ad4c475b766 (127.2.56.2:35533): tablet e1478292dd0340dc8030536ae3d23211 (table test-table [id=4ee297fa1b294525b98ad7d88a699246]) successfully deleted
I20260928 12:21:50.632539 9779 catalog_manager.cc:4841] TS 239e6f2672f64a35b0fa47287b25a451 (127.2.56.3:44701): tablet e1478292dd0340dc8030536ae3d23211 (table test-table [id=4ee297fa1b294525b98ad7d88a699246]) successfully deleted
I20260928 12:21:50.632531 10210 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab8ee3bf68c46ea9c223c16b3cb30e5 (DEFAULT_TABLE table=test-table [id=b32e09eddc68403ca6d871bf14fd8678]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:50.632604 10305 tablet_bootstrap.cc:654] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P cd35f7e208354f2f88192ad4c475b766: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:50.632637 10210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab8ee3bf68c46ea9c223c16b3cb30e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:50.632968 9930 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab8ee3bf68c46ea9c223c16b3cb30e5 (DEFAULT_TABLE table=test-table [id=b32e09eddc68403ca6d871bf14fd8678]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:21:50.633072 9930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab8ee3bf68c46ea9c223c16b3cb30e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:50.633556 9778 catalog_manager.cc:4841] TS e016b1c4cd81472c8432f8f34766874f (127.2.56.1:44449): tablet e1478292dd0340dc8030536ae3d23211 (table test-table [id=4ee297fa1b294525b98ad7d88a699246]) successfully deleted
I20260928 12:21:50.633971 10304 tablet_bootstrap.cc:492] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P 239e6f2672f64a35b0fa47287b25a451: Bootstrap starting.
I20260928 12:21:50.634130 10306 tablet_bootstrap.cc:492] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P e016b1c4cd81472c8432f8f34766874f: Bootstrap starting.
I20260928 12:21:50.634322 10304 tablet_bootstrap.cc:654] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P 239e6f2672f64a35b0fa47287b25a451: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:50.634473 10306 tablet_bootstrap.cc:654] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P e016b1c4cd81472c8432f8f34766874f: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:50.634902 10304 tablet_bootstrap.cc:492] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P 239e6f2672f64a35b0fa47287b25a451: No bootstrap required, opened a new log
I20260928 12:21:50.634949 10305 tablet_bootstrap.cc:492] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P cd35f7e208354f2f88192ad4c475b766: No bootstrap required, opened a new log
I20260928 12:21:50.634951 10304 ts_tablet_manager.cc:1397] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P 239e6f2672f64a35b0fa47287b25a451: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 12:21:50.634997 10305 ts_tablet_manager.cc:1397] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P cd35f7e208354f2f88192ad4c475b766: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 12:21:50.635113 10304 raft_consensus.cc:348] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P 239e6f2672f64a35b0fa47287b25a451 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } }
I20260928 12:21:50.635195 10304 raft_consensus.cc:374] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P 239e6f2672f64a35b0fa47287b25a451 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:50.635166 10305 raft_consensus.cc:348] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P cd35f7e208354f2f88192ad4c475b766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } }
I20260928 12:21:50.635218 10304 raft_consensus.cc:729] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P 239e6f2672f64a35b0fa47287b25a451 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 239e6f2672f64a35b0fa47287b25a451, State: Initialized, Role: FOLLOWER
I20260928 12:21:50.635238 10305 raft_consensus.cc:374] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P cd35f7e208354f2f88192ad4c475b766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:50.635257 10305 raft_consensus.cc:729] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P cd35f7e208354f2f88192ad4c475b766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd35f7e208354f2f88192ad4c475b766, State: Initialized, Role: FOLLOWER
I20260928 12:21:50.635264 10304 consensus_queue.cc:260] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P 239e6f2672f64a35b0fa47287b25a451 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } }
I20260928 12:21:50.635303 10305 consensus_queue.cc:260] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P cd35f7e208354f2f88192ad4c475b766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } }
I20260928 12:21:50.635353 10304 ts_tablet_manager.cc:1428] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P 239e6f2672f64a35b0fa47287b25a451: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:50.635389 10305 ts_tablet_manager.cc:1428] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P cd35f7e208354f2f88192ad4c475b766: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:50.635785 10306 tablet_bootstrap.cc:492] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P e016b1c4cd81472c8432f8f34766874f: No bootstrap required, opened a new log
I20260928 12:21:50.635833 10306 ts_tablet_manager.cc:1397] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P e016b1c4cd81472c8432f8f34766874f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:21:50.635982 10306 raft_consensus.cc:348] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P e016b1c4cd81472c8432f8f34766874f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } }
I20260928 12:21:50.636062 10306 raft_consensus.cc:374] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P e016b1c4cd81472c8432f8f34766874f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:50.636081 10306 raft_consensus.cc:729] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P e016b1c4cd81472c8432f8f34766874f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e016b1c4cd81472c8432f8f34766874f, State: Initialized, Role: FOLLOWER
I20260928 12:21:50.636118 10306 consensus_queue.cc:260] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P e016b1c4cd81472c8432f8f34766874f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } }
I20260928 12:21:50.636216 10306 ts_tablet_manager.cc:1428] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P e016b1c4cd81472c8432f8f34766874f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:50.709684 10379 raft_consensus.cc:482] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P 239e6f2672f64a35b0fa47287b25a451 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:21:50.709800 10379 raft_consensus.cc:504] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P 239e6f2672f64a35b0fa47287b25a451 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } }
I20260928 12:21:50.709969 10379 leader_election.cc:288] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P 239e6f2672f64a35b0fa47287b25a451 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e016b1c4cd81472c8432f8f34766874f (127.2.56.1:44449), cd35f7e208354f2f88192ad4c475b766 (127.2.56.2:35533)
I20260928 12:21:50.710204 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab8ee3bf68c46ea9c223c16b3cb30e5" candidate_uuid: "239e6f2672f64a35b0fa47287b25a451" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd35f7e208354f2f88192ad4c475b766" is_pre_election: true
I20260928 12:21:50.710213 9950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab8ee3bf68c46ea9c223c16b3cb30e5" candidate_uuid: "239e6f2672f64a35b0fa47287b25a451" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e016b1c4cd81472c8432f8f34766874f" is_pre_election: true
I20260928 12:21:50.710318 10088 raft_consensus.cc:2454] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P cd35f7e208354f2f88192ad4c475b766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 239e6f2672f64a35b0fa47287b25a451 in term 0.
I20260928 12:21:50.710322 9950 raft_consensus.cc:2454] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P e016b1c4cd81472c8432f8f34766874f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 239e6f2672f64a35b0fa47287b25a451 in term 0.
I20260928 12:21:50.710507 10164 leader_election.cc:302] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P 239e6f2672f64a35b0fa47287b25a451 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 239e6f2672f64a35b0fa47287b25a451, cd35f7e208354f2f88192ad4c475b766; no voters:
I20260928 12:21:50.710624 10379 raft_consensus.cc:2781] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P 239e6f2672f64a35b0fa47287b25a451 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:21:50.710662 10379 raft_consensus.cc:482] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P 239e6f2672f64a35b0fa47287b25a451 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:21:50.710709 10379 raft_consensus.cc:3037] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P 239e6f2672f64a35b0fa47287b25a451 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:50.711624 10379 raft_consensus.cc:504] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P 239e6f2672f64a35b0fa47287b25a451 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } }
I20260928 12:21:50.711763 10379 leader_election.cc:288] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P 239e6f2672f64a35b0fa47287b25a451 [CANDIDATE]: Term 1 election: Requested vote from peers e016b1c4cd81472c8432f8f34766874f (127.2.56.1:44449), cd35f7e208354f2f88192ad4c475b766 (127.2.56.2:35533)
I20260928 12:21:50.711928 9950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab8ee3bf68c46ea9c223c16b3cb30e5" candidate_uuid: "239e6f2672f64a35b0fa47287b25a451" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e016b1c4cd81472c8432f8f34766874f"
I20260928 12:21:50.712020 9950 raft_consensus.cc:3037] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P e016b1c4cd81472c8432f8f34766874f [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:50.711997 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab8ee3bf68c46ea9c223c16b3cb30e5" candidate_uuid: "239e6f2672f64a35b0fa47287b25a451" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd35f7e208354f2f88192ad4c475b766"
I20260928 12:21:50.712092 10088 raft_consensus.cc:3037] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P cd35f7e208354f2f88192ad4c475b766 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:50.712766 9950 raft_consensus.cc:2454] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P e016b1c4cd81472c8432f8f34766874f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 239e6f2672f64a35b0fa47287b25a451 in term 1.
I20260928 12:21:50.712844 10088 raft_consensus.cc:2454] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P cd35f7e208354f2f88192ad4c475b766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 239e6f2672f64a35b0fa47287b25a451 in term 1.
I20260928 12:21:50.712947 10162 leader_election.cc:302] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P 239e6f2672f64a35b0fa47287b25a451 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 239e6f2672f64a35b0fa47287b25a451, e016b1c4cd81472c8432f8f34766874f; no voters:
I20260928 12:21:50.713034 10379 raft_consensus.cc:2781] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P 239e6f2672f64a35b0fa47287b25a451 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:21:50.713088 10379 raft_consensus.cc:686] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P 239e6f2672f64a35b0fa47287b25a451 [term 1 LEADER]: Becoming Leader. State: Replica: 239e6f2672f64a35b0fa47287b25a451, State: Running, Role: LEADER
I20260928 12:21:50.713143 10379 consensus_queue.cc:237] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P 239e6f2672f64a35b0fa47287b25a451 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } }
I20260928 12:21:50.713807 9793 catalog_manager.cc:5495] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P 239e6f2672f64a35b0fa47287b25a451 reported cstate change: term changed from 0 to 1, leader changed from <none> to 239e6f2672f64a35b0fa47287b25a451 (127.2.56.3). New cstate: current_term: 1 leader_uuid: "239e6f2672f64a35b0fa47287b25a451" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "239e6f2672f64a35b0fa47287b25a451" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 44701 } health_report { overall_health: HEALTHY } } }
I20260928 12:21:50.730847 10088 raft_consensus.cc:1261] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P cd35f7e208354f2f88192ad4c475b766 [term 1 FOLLOWER]: Refusing update from remote peer 239e6f2672f64a35b0fa47287b25a451: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:21:50.730878 9950 raft_consensus.cc:1261] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P e016b1c4cd81472c8432f8f34766874f [term 1 FOLLOWER]: Refusing update from remote peer 239e6f2672f64a35b0fa47287b25a451: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:21:50.731091 10322 consensus_queue.cc:1035] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P 239e6f2672f64a35b0fa47287b25a451 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e016b1c4cd81472c8432f8f34766874f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:21:50.731287 10322 consensus_queue.cc:1035] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P 239e6f2672f64a35b0fa47287b25a451 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd35f7e208354f2f88192ad4c475b766" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:21:50.740625 9792 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:21:50.740754 9792 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:21:50.741415 9792 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 049b783e402e4560b7d3b9b60de4b55b: Sending DeleteTablet for 3 replicas of tablet 1ab8ee3bf68c46ea9c223c16b3cb30e5
I20260928 12:21:50.741671 9930 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ab8ee3bf68c46ea9c223c16b3cb30e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:50 UTC) from {username='oryx', principal='oryx/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:58382
I20260928 12:21:50.741686 10210 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ab8ee3bf68c46ea9c223c16b3cb30e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:50 UTC) from {username='oryx', principal='oryx/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:42046
I20260928 12:21:50.741724 10070 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ab8ee3bf68c46ea9c223c16b3cb30e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:21:50 UTC) from {username='oryx', principal='oryx/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:56496
I20260928 12:21:50.741773 10327 tablet_replica.cc:330] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P e016b1c4cd81472c8432f8f34766874f: stopping tablet replica
I20260928 12:21:50.741832 10327 raft_consensus.cc:2229] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P e016b1c4cd81472c8432f8f34766874f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:21:50.741861 10327 raft_consensus.cc:2258] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P e016b1c4cd81472c8432f8f34766874f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:50.741925 10328 tablet_replica.cc:330] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P 239e6f2672f64a35b0fa47287b25a451: stopping tablet replica
I20260928 12:21:50.741986 10328 raft_consensus.cc:2229] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P 239e6f2672f64a35b0fa47287b25a451 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:21:50.742051 10328 raft_consensus.cc:2258] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P 239e6f2672f64a35b0fa47287b25a451 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:50.742147 10329 tablet_replica.cc:330] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P cd35f7e208354f2f88192ad4c475b766: stopping tablet replica
I20260928 12:21:50.742213 10329 raft_consensus.cc:2229] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P cd35f7e208354f2f88192ad4c475b766 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:21:50.742264 10329 raft_consensus.cc:2258] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P cd35f7e208354f2f88192ad4c475b766 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:21:50.742265 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 9865
I20260928 12:21:50.744037 10329 ts_tablet_manager.cc:1905] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P cd35f7e208354f2f88192ad4c475b766: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:50.744139 10328 ts_tablet_manager.cc:1905] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P 239e6f2672f64a35b0fa47287b25a451: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:21:50.745800 10328 ts_tablet_manager.cc:1918] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P 239e6f2672f64a35b0fa47287b25a451: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:21:50.745868 10328 log.cc:1196] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P 239e6f2672f64a35b0fa47287b25a451: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-2/wal/wals/1ab8ee3bf68c46ea9c223c16b3cb30e5
I20260928 12:21:50.745874 10329 ts_tablet_manager.cc:1918] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P cd35f7e208354f2f88192ad4c475b766: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:21:50.745939 10329 log.cc:1196] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P cd35f7e208354f2f88192ad4c475b766: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598072638673-2272-0/minicluster-data/ts-1/wal/wals/1ab8ee3bf68c46ea9c223c16b3cb30e5
I20260928 12:21:50.746078 10328 ts_tablet_manager.cc:1939] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P 239e6f2672f64a35b0fa47287b25a451: Deleting consensus metadata
I20260928 12:21:50.746126 10329 ts_tablet_manager.cc:1939] T 1ab8ee3bf68c46ea9c223c16b3cb30e5 P cd35f7e208354f2f88192ad4c475b766: Deleting consensus metadata
I20260928 12:21:50.746440 9779 catalog_manager.cc:4841] TS 239e6f2672f64a35b0fa47287b25a451 (127.2.56.3:44701): tablet 1ab8ee3bf68c46ea9c223c16b3cb30e5 (table test-table [id=b32e09eddc68403ca6d871bf14fd8678]) successfully deleted
I20260928 12:21:50.746459 9780 catalog_manager.cc:4841] TS cd35f7e208354f2f88192ad4c475b766 (127.2.56.2:35533): tablet 1ab8ee3bf68c46ea9c223c16b3cb30e5 (table test-table [id=b32e09eddc68403ca6d871bf14fd8678]) successfully deleted
W20260928 12:21:50.750159 9778 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.56.1:44449 (error 108)
I20260928 12:21:50.750419 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 10005
I20260928 12:21:50.751178 10387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:50.750565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:44449 (local address 127.0.0.1:58390)
0928 12:21:50.751051 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:50.751104 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.1:44449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":43,"threads_started":1}
W20260928 12:21:50.751715 9778 catalog_manager.cc:4584] TS e016b1c4cd81472c8432f8f34766874f (127.2.56.1:44449): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1ab8ee3bf68c46ea9c223c16b3cb30e5: Network error: Client connection negotiation failed: client connection to 127.2.56.1:44449: connect: Connection refused (error 111)
I20260928 12:21:50.756084 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 10145
I20260928 12:21:50.761300 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 9761
2026-09-28T12:21:50Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2574 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:21:50 dist-test-slave-920p krb5kdc[10390](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:21:50 dist-test-slave-920p krb5kdc[10390](info): set up 2 sockets
Sep 28 12:21:50 dist-test-slave-920p krb5kdc[10390](info): commencing operation
krb5kdc: starting...
W20260928 12:21:51.714026 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:21:51 dist-test-slave-920p krb5kdc[10390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:21:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:21:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
I20260928 12:21:51.837788 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:43635
--webserver_interface=127.2.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:36263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:43635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:51.924275 10403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:51.924517 10403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:51.924551 10403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:51.925755 10403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:21:51.925792 10403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:51.925809 10403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:51.925825 10403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:21:51.925839 10403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:21:51.927685 10403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:36263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:43635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:43635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:51.927985 10403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:51.928238 10403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:51.933151 10409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:51.933159 10411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:51.933197 10408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:51.933391 10403 server_base.cc:1034] running on GCE node
I20260928 12:21:51.933676 10403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:51.934005 10403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:51.935146 10403 hybrid_clock.cc:648] HybridClock initialized: now 1790598111935134 us; error 29 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T12:21:51Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1183 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:21:52 dist-test-slave-920p krb5kdc[10416](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:21:52 dist-test-slave-920p krb5kdc[10416](info): set up 2 sockets
Sep 28 12:21:52 dist-test-slave-920p krb5kdc[10416](info): commencing operation
krb5kdc: starting...
W20260928 12:21:52.890198 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:21:52 dist-test-slave-920p krb5kdc[10416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:21:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:21:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
I20260928 12:21:53.010614 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:45257
--webserver_interface=127.2.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:46441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:45257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:53.097782 10429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:53.098028 10429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:53.098059 10429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:53.099274 10429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:21:53.099321 10429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:53.099339 10429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:53.099355 10429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:21:53.099368 10429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:21:53.100952 10429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:46441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:45257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:45257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:53.101233 10429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:53.101491 10429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:53.106499 10434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:53.106542 10429 server_base.cc:1034] running on GCE node
W20260928 12:21:53.106511 10437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:53.106518 10435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:53.106925 10429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:53.107218 10429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:53.108369 10429 hybrid_clock.cc:648] HybridClock initialized: now 1790598113108349 us; error 38 us; skew 500 ppm
Sep 28 12:21:53 dist-test-slave-920p krb5kdc[10416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598113, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:53.115578 10429 init.cc:377] Logged in from keytab as kudu/127.2.56.62@KRBTEST.COM (short username kudu)
I20260928 12:21:53.116056 10429 webserver.cc:460] Webserver started at http://127.2.56.62:38823/ using document root <none> and password file <none>
I20260928 12:21:53.116269 10429 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:53.116308 10429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:53.116410 10429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:53.117558 10429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "40b174eb9518407e90d01b98e929f4a4"
format_stamp: "Formatted at 2026-09-28 12:21:53 on dist-test-slave-920p"
server_key: "d91de99440bd03e4c9274d5751791197"
server_key_iv: "950938b32d4cb0f9332cd7998b974a7b"
server_key_version: "encryptionkey@0"
I20260928 12:21:53.117885 10429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "40b174eb9518407e90d01b98e929f4a4"
format_stamp: "Formatted at 2026-09-28 12:21:53 on dist-test-slave-920p"
server_key: "d91de99440bd03e4c9274d5751791197"
server_key_iv: "950938b32d4cb0f9332cd7998b974a7b"
server_key_version: "encryptionkey@0"
I20260928 12:21:53.119144 10429 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:21:53.119834 10444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:53.120009 10429 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:53.120095 10429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "40b174eb9518407e90d01b98e929f4a4"
format_stamp: "Formatted at 2026-09-28 12:21:53 on dist-test-slave-920p"
server_key: "d91de99440bd03e4c9274d5751791197"
server_key_iv: "950938b32d4cb0f9332cd7998b974a7b"
server_key_version: "encryptionkey@0"
I20260928 12:21:53.120152 10429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:53.145434 10429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:53.147863 10429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:53.148026 10429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:53.152437 10429 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:45257
I20260928 12:21:53.152474 10496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:45257 every 8 connection(s)
I20260928 12:21:53.152861 10429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:21:53.153858 10497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:21:53.155718 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 10429
I20260928 12:21:53.155807 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:21:53.156008 2272 external_mini_cluster.cc:1428] Setting key f337c3be6a9729cee30d677d7b533bbd
I20260928 12:21:53.156513 10497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4: Bootstrap starting.
I20260928 12:21:53.157168 10497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4: Neither blocks nor log segments found. Creating new log.
I20260928 12:21:53.157506 10497 log.cc:824] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:53.158252 10497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4: No bootstrap required, opened a new log
I20260928 12:21:53.160373 10497 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b174eb9518407e90d01b98e929f4a4" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 45257 } }
I20260928 12:21:53.160529 10497 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:21:53.160553 10497 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40b174eb9518407e90d01b98e929f4a4, State: Initialized, Role: FOLLOWER
I20260928 12:21:53.160679 10497 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b174eb9518407e90d01b98e929f4a4" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 45257 } }
I20260928 12:21:53.160749 10497 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:21:53.160772 10497 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:21:53.160818 10497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:21:53.161501 10497 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b174eb9518407e90d01b98e929f4a4" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 45257 } }
I20260928 12:21:53.161605 10497 leader_election.cc:302] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40b174eb9518407e90d01b98e929f4a4; no voters:
I20260928 12:21:53.161777 10497 leader_election.cc:288] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:21:53.161814 10502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:21:53.161970 10502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4 [term 1 LEADER]: Becoming Leader. State: Replica: 40b174eb9518407e90d01b98e929f4a4, State: Running, Role: LEADER
I20260928 12:21:53.162074 10497 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:21:53.162140 10502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b174eb9518407e90d01b98e929f4a4" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 45257 } }
I20260928 12:21:53.162523 10502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40b174eb9518407e90d01b98e929f4a4. Latest consensus state: current_term: 1 leader_uuid: "40b174eb9518407e90d01b98e929f4a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b174eb9518407e90d01b98e929f4a4" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 45257 } } }
I20260928 12:21:53.162675 10502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4 [sys.catalog]: This master's current role is: LEADER
Sep 28 12:21:53 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:53.163269 10510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:21:53.163445 10503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40b174eb9518407e90d01b98e929f4a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b174eb9518407e90d01b98e929f4a4" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 45257 } } }
I20260928 12:21:53.163532 10503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4 [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:53.163833 10510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:21:53.166108 10510 catalog_manager.cc:1347] Generated new cluster ID: 05089bde85d245ed9aecfc9506546ac2
I20260928 12:21:53.166160 10510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:21:53.167997 10500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:53.156547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57440 (local address 127.2.56.62:45257)
0928 12:21:53.156709 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:53.156718 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:53.156746 (+ 28us) server_negotiation.cc:407] Connection header received
0928 12:21:53.157219 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:53.157228 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:53.157326 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:53.157460 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:53.158197 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:53.159164 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:53.160261 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:53.160749 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:53.160854 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:53.163717 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:53.163722 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:53.163722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:53.163735 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:53.165980 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:53.166355 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:53.166359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:53.166359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:53.166472 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:53.166676 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:53.166677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:53.166677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:53.166987 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:53.167134 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:53.167632 (+ 498us) server_negotiation.cc:299] Negotiation successful
0928 12:21:53.167767 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":76,"threads_started":1}
I20260928 12:21:53.186113 10510 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:21:53.186969 10510 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:21:53.192466 10510 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4: Generated new TSK 0
I20260928 12:21:53.192679 10510 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab.
Entry for principal kudu/127.2.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab.
I20260928 12:21:53.219914 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.1:0
--local_ip_for_outbound_sockets=127.2.56.1
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:45257
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:46441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:53.307042 10523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:53.307291 10523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:53.307315 10523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:53.308552 10523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:53.308598 10523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:53.308626 10523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.1
I20260928 12:21:53.310184 10523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:46441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:45257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.1
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:53.310462 10523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:53.310703 10523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:53.315985 10528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:53.315982 10531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:53.316010 10529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:53.316221 10523 server_base.cc:1034] running on GCE node
I20260928 12:21:53.316509 10523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:53.316771 10523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:53.317920 10523 hybrid_clock.cc:648] HybridClock initialized: now 1790598113317903 us; error 32 us; skew 500 ppm
Sep 28 12:21:53 dist-test-slave-920p krb5kdc[10416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598113, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:53.320537 10523 init.cc:377] Logged in from keytab as kudu/127.2.56.1@KRBTEST.COM (short username kudu)
I20260928 12:21:53.320993 10523 webserver.cc:460] Webserver started at http://127.2.56.1:35553/ using document root <none> and password file <none>
I20260928 12:21:53.321223 10523 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:53.321296 10523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:53.321436 10523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:53.322613 10523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-0/data/instance:
uuid: "b568d3570a4c46ea93f186d00dd1e108"
format_stamp: "Formatted at 2026-09-28 12:21:53 on dist-test-slave-920p"
server_key: "aa42b2b0310ded9f8216a1558ab225b5"
server_key_iv: "e4769012f0835efc35c6d2db5ee01d14"
server_key_version: "encryptionkey@0"
I20260928 12:21:53.323014 10523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance:
uuid: "b568d3570a4c46ea93f186d00dd1e108"
format_stamp: "Formatted at 2026-09-28 12:21:53 on dist-test-slave-920p"
server_key: "aa42b2b0310ded9f8216a1558ab225b5"
server_key_iv: "e4769012f0835efc35c6d2db5ee01d14"
server_key_version: "encryptionkey@0"
I20260928 12:21:53.324337 10523 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:21:53.325021 10538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:53.325179 10523 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:53.325290 10523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-0/wal
uuid: "b568d3570a4c46ea93f186d00dd1e108"
format_stamp: "Formatted at 2026-09-28 12:21:53 on dist-test-slave-920p"
server_key: "aa42b2b0310ded9f8216a1558ab225b5"
server_key_iv: "e4769012f0835efc35c6d2db5ee01d14"
server_key_version: "encryptionkey@0"
I20260928 12:21:53.325362 10523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:53.338186 10523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:53.340173 10523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:53.340327 10523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:53.340583 10523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:21:53.340924 10523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:53.340958 10523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:53.340991 10523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:53.341012 10523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:53.347321 10523 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.1:43339
I20260928 12:21:53.347381 10651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.1:43339 every 8 connection(s)
I20260928 12:21:53.347723 10523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
Sep 28 12:21:53 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598113, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:53.355602 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 10523
I20260928 12:21:53.355705 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance
I20260928 12:21:53.355880 2272 external_mini_cluster.cc:1428] Setting key 8068989a1b27c7b5a83c8b7fa0980f9f
I20260928 12:21:53.356963 10500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:53.348475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:36011 (local address 127.2.56.62:45257)
0928 12:21:53.348532 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:53.348534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:53.348748 (+ 214us) server_negotiation.cc:407] Connection header received
0928 12:21:53.349345 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:53.349346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:53.349357 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:53.349425 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:53.350420 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:53.350958 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:53.351959 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:53.352162 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:53.352209 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:53.354342 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:53.354344 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:53.354344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:53.354352 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:53.355493 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:53.355914 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:53.355915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:53.355916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:53.355976 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:53.356243 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:53.356245 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:53.356245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:53.356409 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:53.356475 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:53.356768 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 12:21:53.356872 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 12:21:53.357061 10654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:53.348427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.1:36011)
0928 12:21:53.348607 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:53.348653 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:53.349215 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:53.349499 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:53.349503 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:53.350113 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:53.350342 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:53.350347 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:53.351024 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:53.351024 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:53.351897 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:53.351900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:53.352217 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:53.352217 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:53.352321 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:53.352797 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:53.352806 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:53.354245 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:53.355564 (+ 1319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:53.355571 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:53.355574 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:53.355840 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:53.356039 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:53.356040 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:53.356040 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:53.356174 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:53.356474 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:53.356480 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:53.356643 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 12:21:53.356729 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 12:21:53.356858 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":85,"threads_started":1}
I20260928 12:21:53.357581 10652 heartbeater.cc:347] Connected to a master server at 127.2.56.62:45257
I20260928 12:21:53.357679 10652 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:53.357868 10652 heartbeater.cc:511] Master 127.2.56.62:45257 requested a full tablet report, sending...
I20260928 12:21:53.358513 10461 ts_manager.cc:194] Registered new tserver with Master: b568d3570a4c46ea93f186d00dd1e108 (127.2.56.1:43339)
I20260928 12:21:53.359439 10461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.1@KRBTEST.COM'} at 127.2.56.1:36011
WARNING: no policy specified for kudu/127.2.56.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab.
Entry for principal kudu/127.2.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab.
I20260928 12:21:53.380131 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.2:0
--local_ip_for_outbound_sockets=127.2.56.2
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:45257
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:46441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:53.466466 10657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:53.466691 10657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:53.466714 10657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:53.467886 10657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:53.467926 10657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:53.467952 10657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.2
I20260928 12:21:53.469380 10657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:46441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:45257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.2
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:53.469583 10657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:53.469779 10657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:53.474920 10663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:53.475054 10657 server_base.cc:1034] running on GCE node
W20260928 12:21:53.474958 10665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:53.474925 10662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:53.475353 10657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:53.475626 10657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:53.476769 10657 hybrid_clock.cc:648] HybridClock initialized: now 1790598113476759 us; error 30 us; skew 500 ppm
Sep 28 12:21:53 dist-test-slave-920p krb5kdc[10416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598113, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:53.479346 10657 init.cc:377] Logged in from keytab as kudu/127.2.56.2@KRBTEST.COM (short username kudu)
I20260928 12:21:53.479799 10657 webserver.cc:460] Webserver started at http://127.2.56.2:38765/ using document root <none> and password file <none>
I20260928 12:21:53.480015 10657 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:53.480062 10657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:53.480191 10657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:53.481335 10657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-1/data/instance:
uuid: "2e7a63ecf0954304acc0dffecfe56cba"
format_stamp: "Formatted at 2026-09-28 12:21:53 on dist-test-slave-920p"
server_key: "7bf4dc160037433d45b0c8b29d261b9d"
server_key_iv: "51066774c89e3d693f0ba95288e1ec56"
server_key_version: "encryptionkey@0"
I20260928 12:21:53.481684 10657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance:
uuid: "2e7a63ecf0954304acc0dffecfe56cba"
format_stamp: "Formatted at 2026-09-28 12:21:53 on dist-test-slave-920p"
server_key: "7bf4dc160037433d45b0c8b29d261b9d"
server_key_iv: "51066774c89e3d693f0ba95288e1ec56"
server_key_version: "encryptionkey@0"
I20260928 12:21:53.482949 10657 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:21:53.483603 10672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:53.483749 10657 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:53.483834 10657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-1/wal
uuid: "2e7a63ecf0954304acc0dffecfe56cba"
format_stamp: "Formatted at 2026-09-28 12:21:53 on dist-test-slave-920p"
server_key: "7bf4dc160037433d45b0c8b29d261b9d"
server_key_iv: "51066774c89e3d693f0ba95288e1ec56"
server_key_version: "encryptionkey@0"
I20260928 12:21:53.483891 10657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:53.501154 10657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:53.503013 10657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:53.503149 10657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:53.503386 10657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:21:53.503701 10657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:53.503733 10657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:53.503765 10657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:53.503789 10657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:53.509913 10657 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.2:44669
I20260928 12:21:53.509958 10785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.2:44669 every 8 connection(s)
I20260928 12:21:53.510294 10657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
I20260928 12:21:53.515007 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 10657
I20260928 12:21:53.515125 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance
I20260928 12:21:53.515333 2272 external_mini_cluster.cc:1428] Setting key 51def63c2a1d69176f9ae298b70c31b7
Sep 28 12:21:53 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598113, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:53.519882 10500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:53.511036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:37617 (local address 127.2.56.62:45257)
0928 12:21:53.511102 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:53.511104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:53.511294 (+ 190us) server_negotiation.cc:407] Connection header received
0928 12:21:53.511858 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:53.511859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:53.511870 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:53.511916 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:53.512932 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:53.513509 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:53.514530 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:53.514721 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:53.514766 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:53.517127 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:53.517129 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:53.517129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:53.517137 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:53.518616 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:53.519005 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:53.519006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:53.519007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:53.519069 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:53.519265 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:53.519266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:53.519267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:53.519395 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:53.519461 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:53.519698 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 12:21:53.519784 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 12:21:53.520138 10788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:53.511002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.2:37617)
0928 12:21:53.511162 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:53.511208 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:53.511758 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:53.511996 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:53.512000 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:53.512610 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:53.512836 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:53.512842 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:53.513574 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:53.513575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:53.514463 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:53.514465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:53.514773 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:53.514774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:53.514926 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:53.515490 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:53.515503 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:53.517038 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:53.518698 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:53.518704 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:53.518706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:53.518942 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:53.519123 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:53.519124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:53.519124 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:53.519220 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:53.519452 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:53.519455 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:53.519583 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 12:21:53.519664 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 12:21:53.519797 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260928 12:21:53.520862 10786 heartbeater.cc:347] Connected to a master server at 127.2.56.62:45257
I20260928 12:21:53.520982 10786 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:53.521214 10786 heartbeater.cc:511] Master 127.2.56.62:45257 requested a full tablet report, sending...
I20260928 12:21:53.521689 10461 ts_manager.cc:194] Registered new tserver with Master: 2e7a63ecf0954304acc0dffecfe56cba (127.2.56.2:44669)
I20260928 12:21:53.522085 10461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.2@KRBTEST.COM'} at 127.2.56.2:37617
WARNING: no policy specified for kudu/127.2.56.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab.
Entry for principal kudu/127.2.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab.
I20260928 12:21:53.539500 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.3:0
--local_ip_for_outbound_sockets=127.2.56.3
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:45257
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:46441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:53.625708 10791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:53.625943 10791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:53.625975 10791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:53.627260 10791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:53.627307 10791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:53.627343 10791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.3
I20260928 12:21:53.628872 10791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:46441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:45257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.3
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:53.629115 10791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:53.629375 10791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:53.634390 10799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:53.634433 10797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:53.634402 10796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:53.634610 10791 server_base.cc:1034] running on GCE node
I20260928 12:21:53.634799 10791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:53.635056 10791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:53.636205 10791 hybrid_clock.cc:648] HybridClock initialized: now 1790598113636192 us; error 35 us; skew 500 ppm
Sep 28 12:21:53 dist-test-slave-920p krb5kdc[10416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598113, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:53.638926 10791 init.cc:377] Logged in from keytab as kudu/127.2.56.3@KRBTEST.COM (short username kudu)
I20260928 12:21:53.639400 10791 webserver.cc:460] Webserver started at http://127.2.56.3:42477/ using document root <none> and password file <none>
I20260928 12:21:53.639647 10791 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:53.639698 10791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:53.639822 10791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:21:53.641194 10791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-2/data/instance:
uuid: "37a5af3e206546f4a34df7d6908c764a"
format_stamp: "Formatted at 2026-09-28 12:21:53 on dist-test-slave-920p"
server_key: "7a337e92df11729f855b546d81dfe00f"
server_key_iv: "31035d19fa4d7641d4d4cbfb5094beaa"
server_key_version: "encryptionkey@0"
I20260928 12:21:53.641616 10791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance:
uuid: "37a5af3e206546f4a34df7d6908c764a"
format_stamp: "Formatted at 2026-09-28 12:21:53 on dist-test-slave-920p"
server_key: "7a337e92df11729f855b546d81dfe00f"
server_key_iv: "31035d19fa4d7641d4d4cbfb5094beaa"
server_key_version: "encryptionkey@0"
I20260928 12:21:53.642942 10791 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:21:53.643683 10806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:53.643874 10791 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:21:53.643961 10791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-2/wal
uuid: "37a5af3e206546f4a34df7d6908c764a"
format_stamp: "Formatted at 2026-09-28 12:21:53 on dist-test-slave-920p"
server_key: "7a337e92df11729f855b546d81dfe00f"
server_key_iv: "31035d19fa4d7641d4d4cbfb5094beaa"
server_key_version: "encryptionkey@0"
I20260928 12:21:53.644032 10791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:53.665148 10791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:53.667270 10791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:53.667402 10791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:53.667683 10791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:21:53.668040 10791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:53.668074 10791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:53.668107 10791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:53.668130 10791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:53.674602 10791 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.3:33937
I20260928 12:21:53.674660 10919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.3:33937 every 8 connection(s)
I20260928 12:21:53.674995 10791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
Sep 28 12:21:53 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598113, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:21:53.684231 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 10791
I20260928 12:21:53.684242 10500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:53.675756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:50007 (local address 127.2.56.62:45257)
0928 12:21:53.675843 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:53.675846 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:53.676026 (+ 180us) server_negotiation.cc:407] Connection header received
0928 12:21:53.676621 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:53.676623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:53.676637 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:53.676692 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:53.677773 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:53.678325 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:53.679364 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:53.679571 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:53.679628 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:53.681844 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:53.681846 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:53.681846 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:53.681854 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:53.682974 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:53.683338 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:53.683339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:53.683339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:53.683397 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:53.683608 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:53.683608 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:53.683609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:53.683730 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:53.683794 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:53.684050 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 12:21:53.684140 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 12:21:53.684312 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance
I20260928 12:21:53.684441 10922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:53.675727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.3:50007)
0928 12:21:53.675888 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:53.675935 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:53.676520 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:53.676774 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:53.676779 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:53.677449 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:53.677681 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:53.677687 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:53.678407 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:53.678408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:53.679273 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:53.679276 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:53.679637 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:53.679638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:53.679767 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:53.680305 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:53.680315 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:53.681753 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:21:53.683042 (+ 1289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:53.683048 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:53.683050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:53.683266 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:53.683450 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:21:53.683451 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:21:53.683451 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:21:53.683553 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:21:53.683781 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:21:53.683787 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:21:53.683930 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 12:21:53.684024 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 12:21:53.684198 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20260928 12:21:53.684527 2272 external_mini_cluster.cc:1428] Setting key 501954b8f53b58b5af717e47abf5ca25
I20260928 12:21:53.684916 10920 heartbeater.cc:347] Connected to a master server at 127.2.56.62:45257
I20260928 12:21:53.685024 10920 heartbeater.cc:464] Registering TS with master...
I20260928 12:21:53.685211 10920 heartbeater.cc:511] Master 127.2.56.62:45257 requested a full tablet report, sending...
I20260928 12:21:53.685613 10461 ts_manager.cc:194] Registered new tserver with Master: 37a5af3e206546f4a34df7d6908c764a (127.2.56.3:33937)
I20260928 12:21:53.686007 10461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.3@KRBTEST.COM'} at 127.2.56.3:50007
I20260928 12:21:53.696244 2272 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.1.keytab.
Entry for principal oryx/127.2.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.1.keytab.
WARNING: no policy specified for oryx/127.2.56.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.56.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.2.keytab.
Entry for principal oryx/127.2.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.2.keytab.
WARNING: no policy specified for oryx/127.2.56.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.56.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.3.keytab.
Entry for principal oryx/127.2.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.3.keytab.
I20260928 12:21:53.764804 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 10523
I20260928 12:21:53.771634 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 10657
I20260928 12:21:53.775892 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 10791
I20260928 12:21:53.779819 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 10429
I20260928 12:21:53.784725 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:45257
--webserver_interface=127.2.56.62
--webserver_port=38823
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:46441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:45257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:21:53.872581 10929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:53.872834 10929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:53.872857 10929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:53.874076 10929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:21:53.874118 10929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:53.874131 10929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:53.874141 10929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:21:53.874152 10929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:21:53.875627 10929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:46441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:45257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:45257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=38823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:53.875859 10929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:53.876080 10929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:53.881045 10937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:53.881050 10935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:53.881083 10934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:53.881260 10929 server_base.cc:1034] running on GCE node
I20260928 12:21:53.881525 10929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:53.881847 10929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:53.883002 10929 hybrid_clock.cc:648] HybridClock initialized: now 1790598113882992 us; error 34 us; skew 500 ppm
Sep 28 12:21:53 dist-test-slave-920p krb5kdc[10416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598113, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:53.886036 10929 init.cc:377] Logged in from keytab as kudu/127.2.56.62@KRBTEST.COM (short username kudu)
I20260928 12:21:53.886500 10929 webserver.cc:460] Webserver started at http://127.2.56.62:38823/ using document root <none> and password file <none>
I20260928 12:21:53.886709 10929 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:53.886770 10929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:53.888124 10929 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:21:53.888734 10944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:53.888926 10929 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:21:53.889010 10929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "40b174eb9518407e90d01b98e929f4a4"
format_stamp: "Formatted at 2026-09-28 12:21:53 on dist-test-slave-920p"
server_key: "d91de99440bd03e4c9274d5751791197"
server_key_iv: "950938b32d4cb0f9332cd7998b974a7b"
server_key_version: "encryptionkey@0"
I20260928 12:21:53.889457 10929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:53.905740 10929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:53.907697 10929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:53.907835 10929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:53.912137 10929 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:45257
I20260928 12:21:53.912173 10996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:45257 every 8 connection(s)
I20260928 12:21:53.912567 10929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:21:53.914198 10997 sys_catalog.cc:263] Verifying existing consensus state
I20260928 12:21:53.914899 10997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4: Bootstrap starting.
I20260928 12:21:53.919423 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 10929
I20260928 12:21:53.919806 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.1:43339
--local_ip_for_outbound_sockets=127.2.56.1
--tserver_master_addrs=127.2.56.62:45257
--webserver_port=35553
--webserver_interface=127.2.56.1
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:46441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.56.1
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:21:53.925130 10997 log.cc:824] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4: Log is configured to *not* fsync() on all Append() calls
I20260928 12:21:53.927181 10997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:21:53.927470 10997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4: Bootstrap complete.
I20260928 12:21:53.929818 10997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b174eb9518407e90d01b98e929f4a4" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 45257 } }
I20260928 12:21:53.930063 10997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40b174eb9518407e90d01b98e929f4a4, State: Initialized, Role: FOLLOWER
I20260928 12:21:53.930234 10997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b174eb9518407e90d01b98e929f4a4" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 45257 } }
I20260928 12:21:53.930307 10997 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:21:53.930351 10997 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:21:53.930413 10997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4 [term 1 FOLLOWER]: Advancing to term 2
I20260928 12:21:53.931469 10997 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b174eb9518407e90d01b98e929f4a4" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 45257 } }
I20260928 12:21:53.931565 10997 leader_election.cc:302] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40b174eb9518407e90d01b98e929f4a4; no voters:
I20260928 12:21:53.931734 10997 leader_election.cc:288] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 12:21:53.931797 11001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 12:21:53.932009 10997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:21:53.932011 11001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4 [term 2 LEADER]: Becoming Leader. State: Replica: 40b174eb9518407e90d01b98e929f4a4, State: Running, Role: LEADER
I20260928 12:21:53.932163 11001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b174eb9518407e90d01b98e929f4a4" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 45257 } }
I20260928 12:21:53.932436 11002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "40b174eb9518407e90d01b98e929f4a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b174eb9518407e90d01b98e929f4a4" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 45257 } } }
I20260928 12:21:53.932504 11002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4 [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:53.932639 11003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40b174eb9518407e90d01b98e929f4a4. Latest consensus state: current_term: 2 leader_uuid: "40b174eb9518407e90d01b98e929f4a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b174eb9518407e90d01b98e929f4a4" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 45257 } } }
I20260928 12:21:53.932687 11003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4 [sys.catalog]: This master's current role is: LEADER
I20260928 12:21:53.932796 11008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:21:53.933367 11008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:21:53.933514 11008 catalog_manager.cc:1259] Loaded cluster ID: 05089bde85d245ed9aecfc9506546ac2
I20260928 12:21:53.933544 11008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:21:53.934808 11008 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:21:53.934990 11008 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 40b174eb9518407e90d01b98e929f4a4: Loaded TSK: 0
I20260928 12:21:53.935213 11008 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 12:21:54.009064 10999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:54.009331 10999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:54.009357 10999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:54.010613 10999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:54.010658 10999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:54.010685 10999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.1
I20260928 12:21:54.012166 10999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:46441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.1.keytab
--principal=oryx/127.2.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.1:43339
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.56.1
--webserver_port=35553
--tserver_master_addrs=127.2.56.62:45257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.1
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:54.012413 10999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:54.012660 10999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:54.017836 11024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:54.017819 11023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:54.017828 11026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:54.018244 10999 server_base.cc:1034] running on GCE node
I20260928 12:21:54.018457 10999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:54.018699 10999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:54.019850 10999 hybrid_clock.cc:648] HybridClock initialized: now 1790598114019838 us; error 28 us; skew 500 ppm
Sep 28 12:21:54 dist-test-slave-920p krb5kdc[10416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598114, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:54.022531 10999 init.cc:377] Logged in from keytab as oryx/127.2.56.1@KRBTEST.COM (short username oryx)
I20260928 12:21:54.022997 10999 webserver.cc:460] Webserver started at http://127.2.56.1:35553/ using document root <none> and password file <none>
I20260928 12:21:54.023214 10999 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:54.023273 10999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:54.024716 10999 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:21:54.025352 11033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:54.025509 10999 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:21:54.025593 10999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-0/wal
uuid: "b568d3570a4c46ea93f186d00dd1e108"
format_stamp: "Formatted at 2026-09-28 12:21:53 on dist-test-slave-920p"
server_key: "aa42b2b0310ded9f8216a1558ab225b5"
server_key_iv: "e4769012f0835efc35c6d2db5ee01d14"
server_key_version: "encryptionkey@0"
I20260928 12:21:54.026000 10999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:54.039501 10999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:54.041759 10999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:54.041882 10999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:54.042160 10999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:21:54.042507 10999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:54.042539 10999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:54.042562 10999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:54.042573 10999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:54.050195 10999 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.1:43339
I20260928 12:21:54.050316 11146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.1:43339 every 8 connection(s)
I20260928 12:21:54.050635 10999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
I20260928 12:21:54.055907 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 10999
I20260928 12:21:54.056224 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.2:44669
--local_ip_for_outbound_sockets=127.2.56.2
--tserver_master_addrs=127.2.56.62:45257
--webserver_port=38765
--webserver_interface=127.2.56.2
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:46441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.56.2
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 12:21:54 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:21:54 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:21:54.060447 11149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:54.051429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.1:36811)
0928 12:21:54.051637 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:54.051687 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:54.052399 (+ 712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:54.053072 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:54.053076 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:54.053670 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:54.053888 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:54.053894 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:54.055494 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:54.055494 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:54.056472 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:54.056475 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:54.057053 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:54.057054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:54.057185 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:54.057754 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:54.057765 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:54.060219 (+ 2454us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":96,"threads_started":1}
W20260928 12:21:54.060521 11150 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:21:54.051550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:36811 (local address 127.2.56.62:45257)
0928 12:21:54.052617 (+ 1067us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:54.052627 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:54.052670 (+ 43us) server_negotiation.cc:407] Connection header received
0928 12:21:54.052810 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:54.052817 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:54.052876 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:54.053025 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:54.054000 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:54.055411 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:54.056544 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:54.056977 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:54.057076 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:54.060366 (+ 3290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.1:36811: BlockingRecv error: recv got EOF from 127.2.56.1:36811 (error 108)
Metrics: {"server-negotiator.queue_time_us":1012,"thread_start_us":79,"threads_started":1}
W20260928 12:21:54.060702 11147 heartbeater.cc:644] Failed to heartbeat to 127.2.56.62:45257 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.56.62:45257: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Sep 28 12:21:54 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:21:54 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:21:54.065124 11149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:54.060965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.1:58859)
0928 12:21:54.061138 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:54.061149 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:54.061190 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:54.061318 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:54.061319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:54.061448 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:54.061494 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:54.061496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:54.062117 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:54.062118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:54.062635 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:54.062636 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:54.062962 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:54.062963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:54.063024 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:54.063448 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:54.063453 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:54.065065 (+ 1612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":65,"threads_started":1}
W20260928 12:21:54.065255 11150 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:21:54.061132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:58859 (local address 127.2.56.62:45257)
0928 12:21:54.061167 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:54.061169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:54.061178 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:21:54.061240 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:54.061241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:54.061252 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:54.061316 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:54.061558 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:54.062054 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:54.062701 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:54.062908 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:54.062960 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:54.065186 (+ 2226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.1:58859: BlockingRecv error: recv got EOF from 127.2.56.1:58859 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 28 12:21:54 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:21:54 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:21:54.069382 11149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:54.065413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.1:39163)
0928 12:21:54.065457 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:54.065465 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:54.065494 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:54.065702 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:54.065703 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:54.065816 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:54.065841 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:54.065842 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:54.066426 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:54.066427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:54.066931 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:54.066932 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:54.067221 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:54.067222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:54.067270 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:54.067702 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:54.067706 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:54.069314 (+ 1608us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 12:21:54.069438 11150 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:21:54.065579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:39163 (local address 127.2.56.62:45257)
0928 12:21:54.065616 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:54.065618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:54.065623 (+ 5us) server_negotiation.cc:407] Connection header received
0928 12:21:54.065639 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:54.065640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:54.065648 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:54.065687 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:54.065900 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:54.066364 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:54.066995 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:54.067169 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:54.067221 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:54.069375 (+ 2154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.1:39163: BlockingRecv error: recv got EOF from 127.2.56.1:39163 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 12:21:54.069568 11147 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 12:21:54.143669 11151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:54.143905 11151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:54.143927 11151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:54.145115 11151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:54.145154 11151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:54.145181 11151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.2
I20260928 12:21:54.146660 11151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:46441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.2.keytab
--principal=oryx/127.2.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.2:44669
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.56.2
--webserver_port=38765
--tserver_master_addrs=127.2.56.62:45257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.2
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:54.146903 11151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:54.147123 11151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:54.152143 11158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:54.152119 11157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:54.152137 11160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:54.152429 11151 server_base.cc:1034] running on GCE node
I20260928 12:21:54.152606 11151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:54.152832 11151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:54.153968 11151 hybrid_clock.cc:648] HybridClock initialized: now 1790598114153957 us; error 31 us; skew 500 ppm
Sep 28 12:21:54 dist-test-slave-920p krb5kdc[10416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598114, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:54.156476 11151 init.cc:377] Logged in from keytab as oryx/127.2.56.2@KRBTEST.COM (short username oryx)
I20260928 12:21:54.156915 11151 webserver.cc:460] Webserver started at http://127.2.56.2:38765/ using document root <none> and password file <none>
I20260928 12:21:54.157120 11151 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:54.157182 11151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:54.158614 11151 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:21:54.159253 11167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:54.159409 11151 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:21:54.159509 11151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-1/wal
uuid: "2e7a63ecf0954304acc0dffecfe56cba"
format_stamp: "Formatted at 2026-09-28 12:21:53 on dist-test-slave-920p"
server_key: "7bf4dc160037433d45b0c8b29d261b9d"
server_key_iv: "51066774c89e3d693f0ba95288e1ec56"
server_key_version: "encryptionkey@0"
I20260928 12:21:54.159914 11151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:54.185708 11151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:54.187739 11151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:54.187903 11151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:54.188160 11151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:21:54.188490 11151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:54.188526 11151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:54.188558 11151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:54.188581 11151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:54.195164 11151 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.2:44669
I20260928 12:21:54.195211 11280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.2:44669 every 8 connection(s)
I20260928 12:21:54.195609 11151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
I20260928 12:21:54.202284 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 11151
Sep 28 12:21:54 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:21:54.202600 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.3:33937
--local_ip_for_outbound_sockets=127.2.56.3
--tserver_master_addrs=127.2.56.62:45257
--webserver_port=42477
--webserver_interface=127.2.56.3
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:46441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.56.3
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 12:21:54 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 12:21:54.203939 11150 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:21:54.196445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:57333 (local address 127.2.56.62:45257)
0928 12:21:54.196534 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:54.196537 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:54.196765 (+ 228us) server_negotiation.cc:407] Connection header received
0928 12:21:54.197426 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:54.197428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:54.197450 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:54.197520 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:54.198635 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:54.199190 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:54.200274 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:54.200483 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:54.200546 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:54.203868 (+ 3322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.2:57333: BlockingRecv error: recv got EOF from 127.2.56.2:57333 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 12:21:54.203986 11283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:54.196435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.2:57333)
0928 12:21:54.196611 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:54.196654 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:54.197309 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:54.197610 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:54.197614 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:54.198280 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:54.198523 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:54.198529 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:54.199288 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:54.199289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:54.200170 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:54.200173 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:54.200568 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:54.200569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:54.200704 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:54.201329 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:54.201339 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:54.203748 (+ 2409us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":76,"threads_started":1}
W20260928 12:21:54.204221 11281 heartbeater.cc:644] Failed to heartbeat to 127.2.56.62:45257 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.56.62:45257: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Sep 28 12:21:54 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:21:54 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:21:54.208643 11283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:54.204453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.2:59821)
0928 12:21:54.204511 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:54.204524 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:54.204575 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:54.204706 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:54.204707 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:54.204846 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:54.204911 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:54.204913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:54.205594 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:54.205595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:54.206119 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:54.206120 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:54.206438 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:54.206439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:54.206509 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:54.206981 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:54.206985 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:54.208592 (+ 1607us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 12:21:54.208792 11150 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:21:54.204517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:59821 (local address 127.2.56.62:45257)
0928 12:21:54.204561 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:54.204563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:54.204574 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:21:54.204636 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:54.204637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:54.204646 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:54.204693 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:54.204979 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:54.205534 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:54.206193 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:54.206381 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:54.206436 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:54.208723 (+ 2287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.2:59821: BlockingRecv error: recv got EOF from 127.2.56.2:59821 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 28 12:21:54 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:21:54 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:21:54.213131 11283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:54.208992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.2:41865)
0928 12:21:54.209036 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:54.209046 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:54.209083 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:54.209197 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:54.209197 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:54.209351 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:54.209379 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:54.209380 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:54.209959 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:54.209960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:54.210445 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:54.210447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:54.210743 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:54.210744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:54.210790 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:54.211295 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:54.211300 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:54.213065 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 12:21:54.213213 11150 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:21:54.209056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:41865 (local address 127.2.56.62:45257)
0928 12:21:54.209105 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:54.209107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:54.209116 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:21:54.209136 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:54.209137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:54.209146 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:54.209190 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:54.209437 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:54.209894 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:54.210507 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:54.210691 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:54.210742 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:54.213155 (+ 2413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.2:41865: BlockingRecv error: recv got EOF from 127.2.56.2:41865 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260928 12:21:54.213301 11281 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 12:21:54.291880 11284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:21:54.292111 11284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:21:54.292135 11284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:21:54.293394 11284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:21:54.293443 11284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:21:54.293473 11284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.3
I20260928 12:21:54.294937 11284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:46441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/krb5kdc/oryx_127.2.56.3.keytab
--principal=oryx/127.2.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.3:33937
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.56.3
--webserver_port=42477
--tserver_master_addrs=127.2.56.62:45257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.3
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:21:54.295167 11284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:21:54.295406 11284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:21:54.300275 11290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:21:54.300300 11292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:54.300346 11284 server_base.cc:1034] running on GCE node
W20260928 12:21:54.300266 11289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:21:54.300693 11284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:21:54.300935 11284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:21:54.302080 11284 hybrid_clock.cc:648] HybridClock initialized: now 1790598114302062 us; error 30 us; skew 500 ppm
Sep 28 12:21:54 dist-test-slave-920p krb5kdc[10416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598114, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:21:54.304592 11284 init.cc:377] Logged in from keytab as oryx/127.2.56.3@KRBTEST.COM (short username oryx)
I20260928 12:21:54.305016 11284 webserver.cc:460] Webserver started at http://127.2.56.3:42477/ using document root <none> and password file <none>
I20260928 12:21:54.305217 11284 fs_manager.cc:359] Metadata directory not provided
I20260928 12:21:54.305296 11284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:21:54.306661 11284 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:21:54.307183 11299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:54.307329 11284 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:21:54.307422 11284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-2/wal
uuid: "37a5af3e206546f4a34df7d6908c764a"
format_stamp: "Formatted at 2026-09-28 12:21:53 on dist-test-slave-920p"
server_key: "7a337e92df11729f855b546d81dfe00f"
server_key_iv: "31035d19fa4d7641d4d4cbfb5094beaa"
server_key_version: "encryptionkey@0"
I20260928 12:21:54.307762 11284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:21:54.323212 11284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:21:54.324967 11284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:21:54.325104 11284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:21:54.325366 11284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:21:54.325685 11284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:21:54.325718 11284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:54.325749 11284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:21:54.325773 11284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:21:54.332093 11284 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.3:33937
I20260928 12:21:54.332147 11412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.3:33937 every 8 connection(s)
I20260928 12:21:54.332502 11284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
Sep 28 12:21:54 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:21:54.338580 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 11284
Sep 28 12:21:54 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 12:21:54.339918 11150 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:21:54.333280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:42585 (local address 127.2.56.62:45257)
0928 12:21:54.333354 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:54.333356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:54.333577 (+ 221us) server_negotiation.cc:407] Connection header received
0928 12:21:54.334109 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:54.334110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:54.334123 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:54.334173 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:54.335123 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:54.335646 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:54.336604 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:54.336788 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:54.336833 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:54.339863 (+ 3030us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.3:42585: BlockingRecv error: recv got EOF from 127.2.56.3:42585 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 12:21:54.340031 11415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:54.333280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.3:42585)
0928 12:21:54.333449 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:54.333488 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:54.334027 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:54.334239 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:54.334243 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:54.334827 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:54.335056 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:54.335061 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:54.335706 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:54.335707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:54.336543 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:54.336545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:54.336839 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:54.336840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:54.336945 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:54.337440 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:54.337449 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:54.339807 (+ 2358us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":99,"threads_started":1}
W20260928 12:21:54.340240 11413 heartbeater.cc:644] Failed to heartbeat to 127.2.56.62:45257 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.56.62:45257: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Sep 28 12:21:54 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:21:54 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:21:54.344655 11415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:54.340441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.3:52385)
0928 12:21:54.340483 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:54.340491 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:54.340522 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:54.340624 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:54.340624 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:54.340751 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:54.340803 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:54.340805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:54.341626 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:54.341627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:54.342278 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:54.342280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:54.342636 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:54.342636 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:54.342699 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:54.342996 (+ 297us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:54.343000 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:54.344584 (+ 1584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 12:21:54.344713 11150 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:21:54.340484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:52385 (local address 127.2.56.62:45257)
0928 12:21:54.340519 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:54.340521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:54.340528 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:21:54.340571 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:54.340571 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:54.340579 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:54.340620 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:54.340855 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:54.341560 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:54.342340 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:54.342579 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:54.342626 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:54.344667 (+ 2041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.3:52385: BlockingRecv error: recv got EOF from 127.2.56.3:52385 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 12:21:54.346930 11417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:54.339105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57446 (local address 127.2.56.62:45257)
0928 12:21:54.339222 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:54.339224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:54.339231 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:21:54.339276 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:54.339276 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:54.339287 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:54.339342 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:54.340080 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:54.340879 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:54.341906 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:54.342220 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:54.342287 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:54.343440 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:21:54.343444 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:21:54.343444 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:21:54.343455 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:21:54.345249 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:54.345629 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:54.345655 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:54.345655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:54.345873 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:21:54.346133 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:21:54.346134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:21:54.346134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:21:54.346425 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:21:54.346544 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 12:21:54.346679 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 12:21:54.346824 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":97,"thread_start_us":62,"threads_started":1}
Sep 28 12:21:54 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:21:54 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:21:54.349515 11415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:54.345302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.3:55463)
0928 12:21:54.345604 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:54.345615 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:54.345658 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:54.345763 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:54.345764 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:54.345887 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:54.345932 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:54.345934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:54.346590 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:54.346591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:54.347235 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:54.347237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:54.347599 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:54.347600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:54.347639 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:54.347990 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:54.347995 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:54.349469 (+ 1474us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285}
W20260928 12:21:54.349578 11150 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:21:54.345359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:55463 (local address 127.2.56.62:45257)
0928 12:21:54.345404 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:54.345405 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:54.345655 (+ 250us) server_negotiation.cc:407] Connection header received
0928 12:21:54.345706 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:54.345707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:54.345715 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:54.345758 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:54.345992 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:54.346518 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:54.347303 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:54.347500 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:54.347557 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:54.349537 (+ 1980us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.3:55463: BlockingRecv error: recv got EOF from 127.2.56.3:55463 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260928 12:21:54.349637 11413 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 12:21:55 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:21:55 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:21:55.074616 11418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:55.069989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.1:33957)
0928 12:21:55.070142 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:55.070155 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:55.070218 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:55.070344 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:55.070345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:55.070534 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:55.070597 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:55.070599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:55.071274 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:55.071275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:55.071857 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:55.071859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:55.072162 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:55.072163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:55.072202 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:55.072714 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:55.072719 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:55.074533 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
W20260928 12:21:55.074725 11419 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:21:55.070095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:33957 (local address 127.2.56.62:45257)
0928 12:21:55.070228 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:55.070231 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:55.070238 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:21:55.070262 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:55.070263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:55.070273 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:55.070335 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:55.070649 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:55.071204 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:55.071924 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:55.072118 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:55.072161 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:55.074639 (+ 2478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.1:33957: BlockingRecv error: recv got EOF from 127.2.56.1:33957 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
Sep 28 12:21:55 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:21:55 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:21:55.219030 11420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:55.213734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.2:54511)
0928 12:21:55.213935 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:55.213948 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:55.214035 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:55.214186 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:55.214188 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:55.214406 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:55.214471 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:55.214473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:55.215191 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:55.215194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:55.215881 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:55.215885 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:55.216256 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:55.216257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:55.216313 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:55.216880 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:55.216885 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:55.218914 (+ 2029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":86,"threads_started":1}
W20260928 12:21:55.219149 11419 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:21:55.213834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:54511 (local address 127.2.56.62:45257)
0928 12:21:55.213912 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:55.213915 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:55.214018 (+ 103us) server_negotiation.cc:407] Connection header received
0928 12:21:55.214100 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:55.214101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:55.214117 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:55.214171 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:55.214548 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:55.215104 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:55.215974 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:55.216189 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:55.216251 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:55.219084 (+ 2833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.2:54511: BlockingRecv error: recv got EOF from 127.2.56.2:54511 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 12:21:55 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:21:55 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:21:55.354784 11421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:55.350070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.3:48247)
0928 12:21:55.350226 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:55.350239 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:55.350315 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:55.350466 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:55.350467 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:55.350655 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:55.350715 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:55.350717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:55.351383 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:55.351384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:55.351961 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:55.351963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:55.352284 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:55.352286 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:55.352340 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:55.352835 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:55.352840 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:55.354703 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":87,"threads_started":1}
W20260928 12:21:55.354905 11419 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:21:55.350155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:48247 (local address 127.2.56.62:45257)
0928 12:21:55.350231 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:55.350233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:55.350300 (+ 67us) server_negotiation.cc:407] Connection header received
0928 12:21:55.350374 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:55.350374 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:55.350390 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:55.350448 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:55.350781 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:55.351311 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:55.352031 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:55.352226 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:55.352282 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:55.354826 (+ 2544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.3:48247: BlockingRecv error: recv got EOF from 127.2.56.3:48247 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 12:21:56 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:21:56 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:21:56.079844 11422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:56.075087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.1:51995)
0928 12:21:56.075248 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:56.075261 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:56.075331 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:56.075469 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:56.075469 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:56.075657 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:56.075721 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:56.075723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:56.076454 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:56.076455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:56.077018 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:56.077019 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:56.077404 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:56.077405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:56.077448 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:56.077980 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:56.077986 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:56.079769 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":84,"threads_started":1}
W20260928 12:21:56.079947 11423 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:21:56.075164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:51995 (local address 127.2.56.62:45257)
0928 12:21:56.075274 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:56.075276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:56.075313 (+ 37us) server_negotiation.cc:407] Connection header received
0928 12:21:56.075381 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:56.075382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:56.075397 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:56.075458 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:56.075814 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:56.076385 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:56.077099 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:56.077338 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:56.077400 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:56.079870 (+ 2470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.1:51995: BlockingRecv error: recv got EOF from 127.2.56.1:51995 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":57,"threads_started":1}
Sep 28 12:21:56 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:21:56 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:21:56.224617 11424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:56.219752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.2:39057)
0928 12:21:56.219951 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:56.219963 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:56.220043 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:56.220183 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:56.220184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:56.220370 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:56.220434 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:56.220436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:56.221106 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:56.221107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:56.221697 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:56.221699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:56.222149 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:56.222150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:56.222223 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:56.222732 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:56.222737 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:56.224532 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":86,"threads_started":1}
W20260928 12:21:56.224712 11423 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:21:56.219835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:39057 (local address 127.2.56.62:45257)
0928 12:21:56.219931 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:56.219933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:56.220034 (+ 101us) server_negotiation.cc:407] Connection header received
0928 12:21:56.220099 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:56.220101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:56.220114 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:56.220165 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:56.220490 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:56.221040 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:56.221781 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:56.222065 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:56.222126 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:56.224645 (+ 2519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.2:39057: BlockingRecv error: recv got EOF from 127.2.56.2:39057 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 12:21:56 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:21:56 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:21:56.359800 11425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:56.355262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.3:48507)
0928 12:21:56.355412 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:56.355424 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:56.355504 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:56.355640 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:56.355641 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:56.355820 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:56.355885 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:56.355887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:56.356530 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:56.356531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:56.357081 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:56.357083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:56.357442 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:56.357444 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:56.357506 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:56.357970 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:56.357974 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:56.359712 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":71,"threads_started":1}
W20260928 12:21:56.359862 11423 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:21:56.355403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:48507 (local address 127.2.56.62:45257)
0928 12:21:56.355459 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:56.355461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:56.355488 (+ 27us) server_negotiation.cc:407] Connection header received
0928 12:21:56.355553 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:56.355553 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:56.355568 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:56.355614 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:56.355933 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:56.356467 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:56.357137 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:56.357357 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:56.357420 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:56.359800 (+ 2380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.3:48507: BlockingRecv error: recv got EOF from 127.2.56.3:48507 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 28 12:21:57 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:21:57 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:21:57.085065 11426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:57.080417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.1:50199)
0928 12:21:57.080612 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:57.080626 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:57.080708 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:57.080860 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:57.080862 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:57.081075 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:57.081137 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:57.081140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:57.081809 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:57.081810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:57.082381 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:57.082383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:57.082684 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:57.082685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:57.082723 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:57.083186 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:57.083191 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:57.084986 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":88,"threads_started":1}
W20260928 12:21:57.085152 11427 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:21:57.080474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:50199 (local address 127.2.56.62:45257)
0928 12:21:57.080652 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:57.080655 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:57.080681 (+ 26us) server_negotiation.cc:407] Connection header received
0928 12:21:57.080760 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:57.080761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:57.080776 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:57.080845 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:57.081199 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:57.081756 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:57.082446 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:57.082638 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:57.082682 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:57.085076 (+ 2394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.1:50199: BlockingRecv error: recv got EOF from 127.2.56.1:50199 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":53,"threads_started":1}
Sep 28 12:21:57 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:21:57 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:21:57.230170 11428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:57.225205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.2:57303)
0928 12:21:57.225406 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:57.225420 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:57.225508 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:57.225645 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:57.225646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:57.225846 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:57.225917 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:57.225919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:57.226633 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:57.226634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:57.227203 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:57.227205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:57.227535 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:57.227535 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:57.227572 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:57.228035 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:57.228039 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:57.230078 (+ 2039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":119,"threads_started":1}
W20260928 12:21:57.230263 11427 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:21:57.225338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:57303 (local address 127.2.56.62:45257)
0928 12:21:57.225426 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:57.225428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:57.225495 (+ 67us) server_negotiation.cc:407] Connection header received
0928 12:21:57.225566 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:57.225567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:57.225580 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:57.225634 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:57.225977 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:57.226550 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:57.227271 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:57.227481 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:57.227534 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:57.230191 (+ 2657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.2:57303: BlockingRecv error: recv got EOF from 127.2.56.2:57303 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 12:21:57 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:21:57 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:21:57.364964 11429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:57.360284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.3:50229)
0928 12:21:57.360439 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:57.360451 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:57.360520 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:57.360643 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:57.360644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:57.360827 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:57.360889 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:57.360891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:57.361592 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:57.361593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:57.362175 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:57.362178 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:57.362514 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:57.362515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:57.362551 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:57.363025 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:57.363030 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:57.364869 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":72,"threads_started":1}
W20260928 12:21:57.365077 11427 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:21:57.360346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:50229 (local address 127.2.56.62:45257)
0928 12:21:57.360418 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:57.360421 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:57.360510 (+ 89us) server_negotiation.cc:407] Connection header received
0928 12:21:57.360568 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:57.360569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:57.360584 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:57.360631 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:57.360950 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:57.361537 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:57.362253 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:57.362461 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:57.362512 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:57.364983 (+ 2471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.3:50229: BlockingRecv error: recv got EOF from 127.2.56.3:50229 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 28 12:21:58 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:21:58 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:21:58.090503 11430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:58.085590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.1:49641)
0928 12:21:58.085749 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:58.085760 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:58.085820 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:58.085986 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:58.085987 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:58.086217 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:58.086296 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:58.086299 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:58.087052 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:58.087053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:58.087606 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:58.087607 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:58.088000 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:58.088001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:58.088066 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:58.088630 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:58.088635 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:58.090432 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":77,"threads_started":1}
W20260928 12:21:58.090608 11431 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:21:58.085668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:49641 (local address 127.2.56.62:45257)
0928 12:21:58.085814 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:58.085815 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:58.085825 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:21:58.085872 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:58.085872 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:58.085886 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:58.085971 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:58.086373 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:58.086989 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:58.087664 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:58.087931 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:58.087998 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:58.090525 (+ 2527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.1:49641: BlockingRecv error: recv got EOF from 127.2.56.1:49641 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":83,"threads_started":1}
Sep 28 12:21:58 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:21:58 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:21:58.241555 11432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:58.230807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.2:47367)
0928 12:21:58.230988 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:58.231000 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:58.231080 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:58.231221 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:58.231222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:58.231432 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:58.231494 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:58.231496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:58.232366 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:58.232368 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:58.232980 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:58.232982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:58.233411 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:58.233412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:58.233455 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:58.239380 (+ 5925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:58.239387 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:58.241477 (+ 2090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":94,"threads_started":1}
W20260928 12:21:58.241667 11431 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:21:58.230890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:47367 (local address 127.2.56.62:45257)
0928 12:21:58.230965 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:58.230967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:58.231087 (+ 120us) server_negotiation.cc:407] Connection header received
0928 12:21:58.231129 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:58.231130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:58.231145 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:58.231208 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:58.231582 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:58.232271 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:58.233073 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:58.233334 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:58.233404 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:58.241599 (+ 8195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.2:47367: BlockingRecv error: recv got EOF from 127.2.56.2:47367 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 28 12:21:58 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:21:58 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:21:58.370652 11433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:58.365517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.3:45095)
0928 12:21:58.365685 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:58.365695 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:58.365760 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:58.365933 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:58.365934 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:58.366148 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:58.366226 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:58.366229 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:58.367013 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:58.367014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:58.367690 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:58.367693 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:58.368098 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:58.368099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:58.368145 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:58.368650 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:58.368654 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:58.370579 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":93,"threads_started":1}
W20260928 12:21:58.370740 11431 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:21:58.365618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:45095 (local address 127.2.56.62:45257)
0928 12:21:58.365680 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:58.365682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:58.365746 (+ 64us) server_negotiation.cc:407] Connection header received
0928 12:21:58.365809 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:58.365810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:58.365824 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:58.365885 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:58.366360 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:58.366925 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:58.367760 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:58.368031 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:58.368083 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:58.370673 (+ 2590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.3:45095: BlockingRecv error: recv got EOF from 127.2.56.3:45095 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 28 12:21:59 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:21:59 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:21:59.095839 11434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:59.091014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.1:53657)
0928 12:21:59.091190 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:59.091201 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:59.091276 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:59.091423 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:59.091424 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:59.091604 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:59.091667 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:59.091669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:59.092430 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:59.092431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:59.093000 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:59.093003 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:59.093491 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:59.093492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:59.093553 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:59.093991 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:59.093995 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:59.095769 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":105,"threads_started":1}
W20260928 12:21:59.095939 11435 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:21:59.091104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:53657 (local address 127.2.56.62:45257)
0928 12:21:59.091211 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:59.091213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:59.091268 (+ 55us) server_negotiation.cc:407] Connection header received
0928 12:21:59.091328 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:59.091329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:59.091341 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:59.091404 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:59.091723 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:59.092358 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:59.093071 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:59.093410 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:59.093478 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:59.095865 (+ 2387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.1:53657: BlockingRecv error: recv got EOF from 127.2.56.1:53657 (error 108)
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":55,"threads_started":1}
Sep 28 12:21:59 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:21:59 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:21:59.246941 11436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:59.242077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.2:54637)
0928 12:21:59.242251 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:59.242262 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:59.242333 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:59.242474 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:59.242475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:59.242668 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:59.242735 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:59.242737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:59.243413 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:59.243414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:59.243972 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:59.243974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:59.244414 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:59.244415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:59.244485 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:59.244972 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:59.244977 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:59.246851 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":89,"threads_started":1}
W20260928 12:21:59.247028 11435 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:21:59.242203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:54637 (local address 127.2.56.62:45257)
0928 12:21:59.242255 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:59.242257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:59.242324 (+ 67us) server_negotiation.cc:407] Connection header received
0928 12:21:59.242393 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:59.242394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:59.242408 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:59.242447 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:59.242797 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:59.243346 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:59.244047 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:59.244311 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:59.244382 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:59.246968 (+ 2586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.2:54637: BlockingRecv error: recv got EOF from 127.2.56.2:54637 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 28 12:21:59 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:21:59 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:21:59.375952 11437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:21:59.371181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.3:58261)
0928 12:21:59.371362 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 12:21:59.371377 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 12:21:59.371474 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:21:59.371614 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:21:59.371615 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:21:59.371822 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:21:59.371882 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:59.371884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:59.372562 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:59.372564 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:59.373137 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:21:59.373140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:21:59.373526 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:21:59.373527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:21:59.373584 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:59.374030 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:21:59.374035 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:21:59.375869 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":107,"threads_started":1}
W20260928 12:21:59.376048 11435 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:21:59.371312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:58261 (local address 127.2.56.62:45257)
0928 12:21:59.371378 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 12:21:59.371380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:21:59.371463 (+ 83us) server_negotiation.cc:407] Connection header received
0928 12:21:59.371534 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:21:59.371535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:21:59.371552 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:21:59.371600 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:21:59.371957 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:59.372491 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:59.373212 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:21:59.373461 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:21:59.373521 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:21:59.375972 (+ 2451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.3:58261: BlockingRecv error: recv got EOF from 127.2.56.3:58261 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 28 12:22:00 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:22:00 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:22:00.101157 11438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:00.096327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.1:44227)
0928 12:22:00.096486 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:00.096497 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:00.096560 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:00.096721 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:00.096723 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:00.096916 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:00.096983 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:00.096985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:00.097714 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:00.097716 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:00.098290 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:00.098292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:00.098596 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:00.098597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:00.098636 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:00.099136 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:22:00.099141 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:00.101066 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":82,"threads_started":1}
W20260928 12:22:00.101243 11439 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:22:00.096389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:44227 (local address 127.2.56.62:45257)
0928 12:22:00.096507 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:00.096509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:00.096544 (+ 35us) server_negotiation.cc:407] Connection header received
0928 12:22:00.096616 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:00.096617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:00.096630 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:00.096684 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:00.097044 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:00.097646 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:00.098347 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:00.098548 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:00.098593 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:00.101173 (+ 2580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.1:44227: BlockingRecv error: recv got EOF from 127.2.56.1:44227 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":61,"threads_started":1}
Sep 28 12:22:00 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:22:00 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:22:00.251965 11440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:00.247427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.2:35643)
0928 12:22:00.247558 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:00.247569 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:00.247634 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:00.247749 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:00.247750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:00.247954 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:00.248011 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:00.248013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:00.248642 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:00.248643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:00.249223 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:00.249225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:00.249583 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:00.249584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:00.249621 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:00.250137 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:22:00.250142 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:00.251893 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
W20260928 12:22:00.252094 11439 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:22:00.247473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:35643 (local address 127.2.56.62:45257)
0928 12:22:00.247553 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:00.247556 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:00.247615 (+ 59us) server_negotiation.cc:407] Connection header received
0928 12:22:00.247685 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:00.247686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:00.247698 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:00.247738 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:00.248061 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:00.248585 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:00.249324 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:00.249531 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:00.249583 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:00.252023 (+ 2440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.2:35643: BlockingRecv error: recv got EOF from 127.2.56.2:35643 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 12:22:00 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:22:00 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:22:00.381240 11441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:00.376441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.3:35989)
0928 12:22:00.376611 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:00.376623 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:00.376697 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:00.376823 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:00.376824 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:00.377016 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:00.377079 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:00.377081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:00.377788 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:00.377789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:00.378361 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:00.378363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:00.378670 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:00.378670 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:00.378707 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:00.379191 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:22:00.379195 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:00.381166 (+ 1971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":104,"threads_started":1}
W20260928 12:22:00.381349 11439 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:22:00.376520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:35989 (local address 127.2.56.62:45257)
0928 12:22:00.376575 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:00.376577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:00.376683 (+ 106us) server_negotiation.cc:407] Connection header received
0928 12:22:00.376745 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:00.376745 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:00.376759 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:00.376808 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:00.377144 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:00.377722 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:00.378430 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:00.378620 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:00.378670 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:00.381260 (+ 2590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.3:35989: BlockingRecv error: recv got EOF from 127.2.56.3:35989 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 28 12:22:01 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:22:01 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:22:01.106882 11442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:01.101703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.1:38919)
0928 12:22:01.101873 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:01.101885 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:01.101957 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:01.102093 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:01.102094 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:01.102293 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:01.102360 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:01.102363 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:01.103093 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:01.103093 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:01.103745 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:01.103748 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:01.104175 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:01.104176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:01.104228 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:01.104839 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:22:01.104845 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:01.106729 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":90,"threads_started":1}
W20260928 12:22:01.106917 11443 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:22:01.101770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:38919 (local address 127.2.56.62:45257)
0928 12:22:01.101914 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:01.101916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:01.101949 (+ 33us) server_negotiation.cc:407] Connection header received
0928 12:22:01.102006 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:01.102007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:01.102019 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:01.102083 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:01.102413 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:01.103007 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:01.103833 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:01.104109 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:01.104165 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:01.106831 (+ 2666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.1:38919: BlockingRecv error: recv got EOF from 127.2.56.1:38919 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
Sep 28 12:22:01 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:22:01 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:22:01.257153 11444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:01.252468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.2:54985)
0928 12:22:01.252629 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:01.252640 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:01.252706 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:01.252853 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:01.252855 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:01.253047 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:01.253112 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:01.253114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:01.253800 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:01.253801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:01.254365 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:01.254366 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:01.254701 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:01.254702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:01.254739 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:01.255277 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:22:01.255282 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:01.257081 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":84,"threads_started":1}
W20260928 12:22:01.257302 11443 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:22:01.252505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:54985 (local address 127.2.56.62:45257)
0928 12:22:01.252592 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:01.252594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:01.252690 (+ 96us) server_negotiation.cc:407] Connection header received
0928 12:22:01.252761 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:01.252762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:01.252799 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:01.252839 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:01.253170 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:01.253722 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:01.254442 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:01.254648 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:01.254700 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:01.257203 (+ 2503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.2:54985: BlockingRecv error: recv got EOF from 127.2.56.2:54985 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 12:22:01 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:22:01 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:22:01.386472 11445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:01.381778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.3:33689)
0928 12:22:01.381931 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:01.381944 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:01.382017 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:01.382158 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:01.382160 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:01.382378 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:01.382443 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:01.382445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:01.383106 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:01.383107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:01.383668 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:01.383670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:01.383992 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:01.383992 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:01.384034 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:01.384550 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:22:01.384555 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:01.386387 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
W20260928 12:22:01.386564 11443 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:22:01.381836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:33689 (local address 127.2.56.62:45257)
0928 12:22:01.381898 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:01.381901 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:01.381996 (+ 95us) server_negotiation.cc:407] Connection header received
0928 12:22:01.382069 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:01.382070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:01.382087 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:01.382142 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:01.382503 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:01.383040 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:01.383731 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:01.383936 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:01.383991 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:01.386497 (+ 2506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.3:33689: BlockingRecv error: recv got EOF from 127.2.56.3:33689 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 12:22:02 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:22:02 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:22:02.112458 11446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:02.107465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.1:40449)
0928 12:22:02.107619 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:02.107631 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:02.107690 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:02.107847 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:02.107848 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:02.108058 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:02.108119 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:02.108122 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:02.108838 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:02.108839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:02.109431 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:02.109433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:02.109748 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:02.109749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:02.109802 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:02.110338 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:22:02.110343 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:02.112352 (+ 2009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
W20260928 12:22:02.112525 11447 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:22:02.107578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:40449 (local address 127.2.56.62:45257)
0928 12:22:02.107706 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:02.107708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:02.107720 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:22:02.107746 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:02.107747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:02.107761 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:02.107835 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:02.108182 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:02.108746 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:02.109494 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:02.109685 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:02.109731 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:02.112454 (+ 2723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.1:40449: BlockingRecv error: recv got EOF from 127.2.56.1:40449 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
Sep 28 12:22:02 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:22:02 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:22:02.262877 11448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:02.257737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.2:36185)
0928 12:22:02.257909 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:02.257920 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:02.257992 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:02.258125 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:02.258126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:02.258327 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:02.258387 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:02.258389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:02.259394 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:02.259395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:02.259958 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:02.259960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:02.260315 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:02.260316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:02.260368 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:02.260855 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:22:02.260860 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:02.262769 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":81,"threads_started":1}
W20260928 12:22:02.262945 11447 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:22:02.257792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:36185 (local address 127.2.56.62:45257)
0928 12:22:02.257854 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:02.257856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:02.257980 (+ 124us) server_negotiation.cc:407] Connection header received
0928 12:22:02.258042 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:02.258043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:02.258055 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:02.258109 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:02.258448 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:02.259309 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:02.260029 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:02.260229 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:02.260281 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:02.262876 (+ 2595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.2:36185: BlockingRecv error: recv got EOF from 127.2.56.2:36185 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 28 12:22:02 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:22:02 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:22:02.391953 11449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:02.387052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.3:48149)
0928 12:22:02.387218 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:02.387229 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:02.387307 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:02.387449 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:02.387450 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:02.387651 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:02.387713 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:02.387715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:02.388413 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:02.388415 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:02.389081 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:02.389085 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:02.389425 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:02.389425 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:02.389467 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:02.389918 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:22:02.389923 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:02.391837 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":85,"threads_started":1}
W20260928 12:22:02.392036 11447 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:22:02.387135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:48149 (local address 127.2.56.62:45257)
0928 12:22:02.387191 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:02.387194 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:02.387292 (+ 98us) server_negotiation.cc:407] Connection header received
0928 12:22:02.387373 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:02.387374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:02.387390 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:02.387433 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:02.387767 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:02.388319 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:02.389149 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:02.389367 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:02.389418 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:02.391971 (+ 2553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.3:48149: BlockingRecv error: recv got EOF from 127.2.56.3:48149 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 28 12:22:03 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:22:03 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:22:03.118027 11450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:03.113039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.1:56767)
0928 12:22:03.113229 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:03.113242 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:03.113355 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:03.113530 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:03.113531 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:03.113737 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:03.113812 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:03.113814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:03.114515 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:03.114517 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:03.115112 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:03.115115 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:03.115467 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:03.115468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:03.115526 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:03.115990 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:22:03.115995 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:03.117935 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":102,"threads_started":1}
W20260928 12:22:03.118115 11451 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:22:03.113119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:56767 (local address 127.2.56.62:45257)
0928 12:22:03.113248 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:03.113249 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:03.113329 (+ 80us) server_negotiation.cc:407] Connection header received
0928 12:22:03.113422 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:03.113423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:03.113440 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:03.113499 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:03.113891 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:03.114439 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:03.115195 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:03.115407 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:03.115464 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:03.118046 (+ 2582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.1:56767: BlockingRecv error: recv got EOF from 127.2.56.1:56767 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":68,"threads_started":1}
Sep 28 12:22:03 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:22:03 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:22:03.274724 11452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:03.263393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.2:35107)
0928 12:22:03.263551 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:03.263564 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:03.263643 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:03.263778 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:03.263779 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:03.263977 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:03.264040 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:03.264042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:03.264700 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:03.264700 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:03.265479 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:03.265484 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:03.265807 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:03.265807 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:03.265844 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:03.272525 (+ 6681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:22:03.272531 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:03.274637 (+ 2106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":81,"threads_started":1}
W20260928 12:22:03.274837 11451 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:22:03.263456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:35107 (local address 127.2.56.62:45257)
0928 12:22:03.263565 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:03.263567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:03.263637 (+ 70us) server_negotiation.cc:407] Connection header received
0928 12:22:03.263701 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:03.263702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:03.263715 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:03.263774 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:03.264091 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:03.264625 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:03.265541 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:03.265744 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:03.265805 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:03.274748 (+ 8943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.2:35107: BlockingRecv error: recv got EOF from 127.2.56.2:35107 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 12:22:03 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:22:03 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:22:03.397421 11453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:03.392619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.3:57623)
0928 12:22:03.392810 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:03.392821 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:03.392897 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:03.393029 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:03.393030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:03.393217 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:03.393326 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:03.393330 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:03.393994 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:03.393994 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:03.394543 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:03.394545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:03.394922 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:03.394923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:03.394977 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:03.395482 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:22:03.395488 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:03.397324 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":129,"threads_started":1}
W20260928 12:22:03.397497 11451 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:22:03.392689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:57623 (local address 127.2.56.62:45257)
0928 12:22:03.392760 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:03.392762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:03.392890 (+ 128us) server_negotiation.cc:407] Connection header received
0928 12:22:03.392960 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:03.392960 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:03.392975 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:03.393016 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:03.393392 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:03.393938 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:03.394625 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:03.394855 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:03.394924 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:03.397434 (+ 2510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.3:57623: BlockingRecv error: recv got EOF from 127.2.56.3:57623 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 28 12:22:04 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:22:04 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:22:04.123625 11454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:04.118584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.1:50781)
0928 12:22:04.118759 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:04.118772 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:04.118844 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:04.118993 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:04.118994 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:04.119187 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:04.119250 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:04.119252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:04.119987 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:04.119988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:04.120581 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:04.120583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:04.121032 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:04.121034 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:04.121116 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:04.121698 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:22:04.121704 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:04.123543 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":96,"threads_started":1}
W20260928 12:22:04.123706 11455 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:22:04.118654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:50781 (local address 127.2.56.62:45257)
0928 12:22:04.118833 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:04.118836 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:04.118846 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:22:04.118902 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:04.118903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:04.118916 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:04.118980 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:04.119315 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:04.119905 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:04.120659 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:04.120963 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:04.121030 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:04.123638 (+ 2608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.1:50781: BlockingRecv error: recv got EOF from 127.2.56.1:50781 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":70,"threads_started":1}
Sep 28 12:22:04 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:22:04 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:22:04.280068 11456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:04.275259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.2:43839)
0928 12:22:04.275476 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:04.275487 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:04.275557 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:04.275696 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:04.275697 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:04.275906 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:04.275970 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:04.275972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:04.276655 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:04.276657 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:04.277210 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:04.277212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:04.277575 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:04.277576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:04.277633 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:04.278094 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:22:04.278099 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:04.279997 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":110,"threads_started":1}
W20260928 12:22:04.280162 11455 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:22:04.275331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:43839 (local address 127.2.56.62:45257)
0928 12:22:04.275396 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:04.275398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:04.275551 (+ 153us) server_negotiation.cc:407] Connection header received
0928 12:22:04.275617 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:04.275618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:04.275633 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:04.275683 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:04.276039 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:04.276583 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:04.277307 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:04.277516 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:04.277572 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:04.280097 (+ 2525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.2:43839: BlockingRecv error: recv got EOF from 127.2.56.2:43839 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 28 12:22:04 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:22:04 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:22:04.402777 11457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:04.397928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.3:45999)
0928 12:22:04.398131 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:04.398146 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:04.398232 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:04.398399 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:04.398400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:04.398610 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:04.398670 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:04.398673 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:04.399401 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:04.399402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:04.399980 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:04.399982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:04.400324 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:04.400326 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:04.400385 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:04.400883 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:22:04.400887 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:04.402685 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":100,"threads_started":1}
W20260928 12:22:04.402876 11455 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:22:04.398099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:45999 (local address 127.2.56.62:45257)
0928 12:22:04.398167 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:04.398170 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:04.398208 (+ 38us) server_negotiation.cc:407] Connection header received
0928 12:22:04.398318 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:04.398319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:04.398334 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:04.398383 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:04.398773 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:04.399315 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:04.400064 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:04.400264 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:04.400320 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:04.402810 (+ 2490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.3:45999: BlockingRecv error: recv got EOF from 127.2.56.3:45999 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 12:22:05 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:22:05 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:22:05.129186 11458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:05.124112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.1:49697)
0928 12:22:05.124313 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:05.124326 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:05.124410 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:05.124559 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:05.124560 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:05.124771 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:05.124834 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:05.124836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:05.125598 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:05.125599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:05.126162 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:05.126164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:05.126511 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:05.126512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:05.126583 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:05.127089 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:22:05.127093 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:05.129101 (+ 2008us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":128,"threads_started":1}
W20260928 12:22:05.129380 11459 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:22:05.124184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:49697 (local address 127.2.56.62:45257)
0928 12:22:05.124317 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:05.124320 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:05.124379 (+ 59us) server_negotiation.cc:407] Connection header received
0928 12:22:05.124475 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:05.124476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:05.124489 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:05.124544 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:05.124922 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:05.125518 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:05.126231 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:05.126452 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:05.126508 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:05.129255 (+ 2747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.1:49697: BlockingRecv error: recv got EOF from 127.2.56.1:49697 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":59,"threads_started":1}
Sep 28 12:22:05 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:22:05 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:22:05.286532 11460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:05.280606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.2:39999)
0928 12:22:05.280761 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:05.280772 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:05.280851 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:05.281014 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:05.281015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:05.281252 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:05.281356 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:05.281359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:05.282325 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:05.282326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:05.282964 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:05.282967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:05.283430 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:05.283431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:05.283494 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:05.284072 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:22:05.284078 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:05.286399 (+ 2321us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":81,"threads_started":1}
W20260928 12:22:05.286574 11459 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:22:05.280665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:39999 (local address 127.2.56.62:45257)
0928 12:22:05.280732 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:05.280735 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:05.280834 (+ 99us) server_negotiation.cc:407] Connection header received
0928 12:22:05.280928 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:05.280929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:05.280944 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:05.281003 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:05.281447 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:05.282238 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:05.283041 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:05.283353 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:05.283433 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:05.286519 (+ 3086us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.2:39999: BlockingRecv error: recv got EOF from 127.2.56.2:39999 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 28 12:22:05 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:22:05 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:22:05.408293 11461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:05.403374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.3:35631)
0928 12:22:05.403529 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:05.403544 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:05.403633 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:05.403782 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:05.403783 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:05.404005 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:05.404089 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:05.404092 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:05.404794 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:05.404795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:05.405454 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:05.405457 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:05.405777 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:05.405778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:05.405844 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:05.406354 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:22:05.406359 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:05.408181 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":86,"threads_started":1}
W20260928 12:22:05.408373 11459 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:22:05.403489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:35631 (local address 127.2.56.62:45257)
0928 12:22:05.403560 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:05.403563 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:05.403617 (+ 54us) server_negotiation.cc:407] Connection header received
0928 12:22:05.403700 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:05.403701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:05.403719 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:05.403768 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:05.404167 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:05.404719 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:05.405524 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:05.405719 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:05.405776 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:05.408308 (+ 2532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.3:35631: BlockingRecv error: recv got EOF from 127.2.56.3:35631 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 12:22:06 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:22:06 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:22:06.134833 11462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:06.129652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.1:45393)
0928 12:22:06.129817 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:06.129830 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:06.129893 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:06.130026 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:06.130027 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:06.130218 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:06.130281 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:06.130283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:06.131018 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:06.131018 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:06.131687 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:06.131690 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:06.132071 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:06.132072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:06.132140 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:06.132723 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:22:06.132729 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:06.134761 (+ 2032us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":105,"threads_started":1}
W20260928 12:22:06.134976 11463 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:22:06.129722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:45393 (local address 127.2.56.62:45257)
0928 12:22:06.129883 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:06.129885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:06.129894 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:22:06.129939 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:06.129940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:06.129952 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:06.130020 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:06.130369 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:06.130949 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:06.131758 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:06.131999 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:06.132068 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:06.134892 (+ 2824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.1:45393: BlockingRecv error: recv got EOF from 127.2.56.1:45393 (error 108)
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":60,"threads_started":1}
Sep 28 12:22:06 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:22:06 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:22:06.291592 11464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:06.287007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.2:36641)
0928 12:22:06.287152 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:06.287166 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:06.287239 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:06.287368 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:06.287369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:06.287571 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:06.287659 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:06.287661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:06.288313 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:06.288314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:06.288865 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:06.288866 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:06.289202 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:06.289203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:06.289294 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:06.289766 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:22:06.289770 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:06.291497 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
W20260928 12:22:06.291667 11463 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:22:06.287109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:36641 (local address 127.2.56.62:45257)
0928 12:22:06.287179 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:06.287181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:06.287228 (+ 47us) server_negotiation.cc:407] Connection header received
0928 12:22:06.287296 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:06.287297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:06.287309 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:06.287355 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:06.287708 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:06.288257 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:06.288932 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:06.289132 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:06.289186 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:06.291599 (+ 2413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.2:36641: BlockingRecv error: recv got EOF from 127.2.56.2:36641 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 12:22:06 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:22:06 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:22:06.413857 11465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:06.408911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.3:58979)
0928 12:22:06.409054 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:06.409069 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:06.409213 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:06.409363 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:06.409364 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:06.409569 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:06.409633 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:06.409636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:06.410375 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:06.410376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:06.410989 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:06.410992 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:06.411326 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:06.411327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:06.411372 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:06.411855 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:22:06.411860 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:06.413762 (+ 1902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":73,"threads_started":1}
W20260928 12:22:06.414011 11463 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:22:06.409018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:58979 (local address 127.2.56.62:45257)
0928 12:22:06.409074 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:06.409076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:06.409182 (+ 106us) server_negotiation.cc:407] Connection header received
0928 12:22:06.409294 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:06.409295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:06.409309 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:06.409353 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:06.409690 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:06.410278 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:06.411059 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:06.411265 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:06.411312 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:06.413893 (+ 2581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.3:58979: BlockingRecv error: recv got EOF from 127.2.56.3:58979 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 28 12:22:07 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:22:07 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:22:07.140130 11466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:07.135338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.1:55283)
0928 12:22:07.135499 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:07.135511 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:07.135577 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:07.135714 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:07.135715 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:07.135898 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:07.135963 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:07.135965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:07.136649 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:07.136651 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:07.137227 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:07.137229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:07.137634 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:07.137634 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:07.137675 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:07.138144 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:22:07.138148 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:07.140060 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":78,"threads_started":1}
W20260928 12:22:07.140223 11467 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:22:07.135390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:55283 (local address 127.2.56.62:45257)
0928 12:22:07.135514 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:07.135516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:07.135569 (+ 53us) server_negotiation.cc:407] Connection header received
0928 12:22:07.135635 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:07.135636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:07.135648 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:07.135701 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:07.136017 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:07.136576 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:07.137356 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:07.137579 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:07.137632 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:07.140151 (+ 2519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.1:55283: BlockingRecv error: recv got EOF from 127.2.56.1:55283 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":64,"threads_started":1}
Sep 28 12:22:07 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:22:07 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:22:07.296667 11468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:07.292043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.2:47839)
0928 12:22:07.292187 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:07.292199 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:07.292259 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:07.292381 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:07.292382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:07.292602 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:07.292669 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:07.292671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:07.293376 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:07.293378 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:07.293945 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:07.293947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:07.294272 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:07.294272 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:07.294309 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:07.294835 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:22:07.294839 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:07.296581 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":86,"threads_started":1}
W20260928 12:22:07.296729 11467 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:22:07.292141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:47839 (local address 127.2.56.62:45257)
0928 12:22:07.292195 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:07.292197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:07.292244 (+ 47us) server_negotiation.cc:407] Connection header received
0928 12:22:07.292307 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:07.292307 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:07.292319 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:07.292369 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:07.292720 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:07.293312 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:07.294017 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:07.294219 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:07.294282 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:07.296672 (+ 2390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.2:47839: BlockingRecv error: recv got EOF from 127.2.56.2:47839 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 28 12:22:07 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:22:07 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:22:07.419354 11469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:07.414394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.3:46377)
0928 12:22:07.414568 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:07.414579 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:07.414649 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:07.414813 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:07.414814 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:07.415019 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:07.415090 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:07.415092 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:07.415771 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:07.415772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:07.416365 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:07.416367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:07.416712 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:07.416713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:07.416772 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:07.417306 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:22:07.417310 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:07.419265 (+ 1955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":105,"threads_started":1}
W20260928 12:22:07.419458 11467 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:22:07.414460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:46377 (local address 127.2.56.62:45257)
0928 12:22:07.414520 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:07.414522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:07.414634 (+ 112us) server_negotiation.cc:407] Connection header received
0928 12:22:07.414733 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:07.414734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:07.414748 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:07.414795 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:07.415142 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:07.415690 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:07.416446 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:07.416634 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:07.416685 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:07.419378 (+ 2693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.3:46377: BlockingRecv error: recv got EOF from 127.2.56.3:46377 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 28 12:22:08 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:22:08 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:22:08.145511 11470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:08.140677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.1:34931)
0928 12:22:08.140844 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:08.140858 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:08.140944 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:08.141107 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:08.141109 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:08.141344 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:08.141413 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:08.141416 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:08.142102 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:08.142102 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:08.142674 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:08.142675 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:08.143039 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:08.143040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:08.143086 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:08.143616 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:22:08.143621 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:08.145439 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":88,"threads_started":1}
W20260928 12:22:08.145633 11471 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:22:08.140744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:34931 (local address 127.2.56.62:45257)
0928 12:22:08.140878 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:08.140880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:08.140932 (+ 52us) server_negotiation.cc:407] Connection header received
0928 12:22:08.141007 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:08.141008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:08.141024 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:08.141093 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:08.141478 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:08.142041 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:08.142755 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:08.142975 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:08.143033 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:08.145540 (+ 2507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.1:34931: BlockingRecv error: recv got EOF from 127.2.56.1:34931 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":69,"threads_started":1}
Sep 28 12:22:08 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:22:08 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:22:08.306600 11472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:08.297193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.2:37463)
0928 12:22:08.297368 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:08.297381 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:08.297458 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:08.297595 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:08.297597 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:08.297796 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:08.297860 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:08.297863 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:08.298539 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:08.298541 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:08.299094 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:08.299096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:08.299421 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:08.299423 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:08.299481 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:08.304560 (+ 5079us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:22:08.304566 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:08.306506 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":111,"threads_started":1}
W20260928 12:22:08.306687 11471 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:22:08.297287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:37463 (local address 127.2.56.62:45257)
0928 12:22:08.297348 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:08.297350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:08.297440 (+ 90us) server_negotiation.cc:407] Connection header received
0928 12:22:08.297518 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:08.297519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:08.297535 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:08.297583 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:08.297931 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:08.298465 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:08.299162 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:08.299363 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:08.299419 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:08.306626 (+ 7207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.2:37463: BlockingRecv error: recv got EOF from 127.2.56.2:37463 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 12:22:08 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:22:08 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.3@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:22:08.424584 11473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:08.419844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.3:38607)
0928 12:22:08.419996 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:08.420008 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:08.420073 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:08.420183 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:08.420183 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:08.420367 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:08.420431 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:08.420433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:08.421093 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:08.421093 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:08.421683 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:08.421686 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:08.422035 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:08.422037 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:08.422092 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:08.422685 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:22:08.422691 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:08.424509 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
W20260928 12:22:08.424682 11471 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:22:08.419898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:38607 (local address 127.2.56.62:45257)
0928 12:22:08.419950 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:08.419953 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:08.420059 (+ 106us) server_negotiation.cc:407] Connection header received
0928 12:22:08.420124 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:08.420124 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:08.420138 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:08.420175 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:08.420481 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:08.421037 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:08.421746 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:08.421944 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:08.421994 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:08.424614 (+ 2620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.3:38607: BlockingRecv error: recv got EOF from 127.2.56.3:38607 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 28 12:22:09 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:22:09 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.1@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:22:09.150929 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:09.146043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.1:55315)
0928 12:22:09.146214 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:09.146224 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:09.146281 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:09.146478 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:09.146478 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:09.146663 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:09.146726 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:09.146727 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:09.147458 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:09.147459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:09.148048 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:09.148050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:09.148378 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:09.148379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:09.148418 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:09.148916 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:22:09.148921 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:09.150840 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":94,"threads_started":1}
W20260928 12:22:09.151006 11475 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:22:09.146161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:55315 (local address 127.2.56.62:45257)
0928 12:22:09.146320 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:09.146323 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:09.146331 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:22:09.146357 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:09.146358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:09.146371 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:09.146467 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:09.146827 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:09.147377 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:09.148119 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:09.148327 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:09.148379 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:09.150932 (+ 2553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.1:55315: BlockingRecv error: recv got EOF from 127.2.56.1:55315 (error 108)
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":79,"threads_started":1}
Sep 28 12:22:09 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:22:09 dist-test-slave-920p krb5kdc[10416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.56.2@KRBTEST.COM for oryx/127.2.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:22:09.311791 11476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:09.307201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45257 (local address 127.2.56.2:34359)
0928 12:22:09.307358 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:09.307369 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:09.307440 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:09.307561 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:09.307562 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:09.307749 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:09.307811 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:09.307813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:09.308525 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:09.308525 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:09.309074 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:09.309076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:09.309428 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:09.309428 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:09.309467 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:09.309945 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:22:09.309950 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:09.311709 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.56.62:45257: Server oryx/127.2.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":93,"threads_started":1}
W20260928 12:22:09.311908 11475 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:22:09.307262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:34359 (local address 127.2.56.62:45257)
0928 12:22:09.307324 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:09.307326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:09.307431 (+ 105us) server_negotiation.cc:407] Connection header received
0928 12:22:09.307489 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:09.307490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:09.307502 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:09.307550 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:09.307893 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:09.308458 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:09.309149 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:09.309374 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:09.309430 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:09.311826 (+ 2396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.56.2:34359: BlockingRecv error: recv got EOF from 127.2.56.2:34359 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 12:22:09.344287 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 10999
I20260928 12:22:09.349172 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 11151
I20260928 12:22:09.353224 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 11284
I20260928 12:22:09.357335 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 10929
2026-09-28T12:22:09Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17410 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T12:22:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:22:09Z Disabled control of system clock
I20260928 12:22:09.384344 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:41069
--webserver_interface=127.2.56.62
--webserver_port=0
--builtin_ntp_servers=127.2.56.20:34117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:41069
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 12:22:09.470954 11480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:22:09.471196 11480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:22:09.471225 11480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:22:09.472445 11480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:22:09.472484 11480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:22:09.472501 11480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:22:09.472517 11480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:22:09.472530 11480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:22:09.473935 11480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:34117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:41069
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:41069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:22:09.474205 11480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:22:09.474509 11480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:22:09.479408 11485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:22:09.479431 11486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:22:09.479445 11488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:22:09.479570 11480 server_base.cc:1034] running on GCE node
I20260928 12:22:09.479913 11480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:22:09.480203 11480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:22:09.481323 11480 hybrid_clock.cc:648] HybridClock initialized: now 1790598129481307 us; error 29 us; skew 500 ppm
I20260928 12:22:09.482565 11480 webserver.cc:460] Webserver started at http://127.2.56.62:36523/ using document root <none> and password file <none>
I20260928 12:22:09.482800 11480 fs_manager.cc:359] Metadata directory not provided
I20260928 12:22:09.482851 11480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:22:09.482986 11480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:22:09.484117 11480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "b6d8c9dd92d049a19e3f02ce35a03894"
format_stamp: "Formatted at 2026-09-28 12:22:09 on dist-test-slave-920p"
server_key: "4b62a97db9318f44532707da16651396"
server_key_iv: "9f1acb256004dfa81eed66bdc99eb9ba"
server_key_version: "encryptionkey@0"
I20260928 12:22:09.484479 11480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "b6d8c9dd92d049a19e3f02ce35a03894"
format_stamp: "Formatted at 2026-09-28 12:22:09 on dist-test-slave-920p"
server_key: "4b62a97db9318f44532707da16651396"
server_key_iv: "9f1acb256004dfa81eed66bdc99eb9ba"
server_key_version: "encryptionkey@0"
I20260928 12:22:09.485859 11480 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:22:09.486553 11494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:09.486686 11480 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:22:09.486771 11480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "b6d8c9dd92d049a19e3f02ce35a03894"
format_stamp: "Formatted at 2026-09-28 12:22:09 on dist-test-slave-920p"
server_key: "4b62a97db9318f44532707da16651396"
server_key_iv: "9f1acb256004dfa81eed66bdc99eb9ba"
server_key_version: "encryptionkey@0"
I20260928 12:22:09.486831 11480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:22:09.500021 11480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:22:09.500361 11480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:22:09.500496 11480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:22:09.504768 11480 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:41069
I20260928 12:22:09.504791 11546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:41069 every 8 connection(s)
I20260928 12:22:09.505155 11480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:22:09.505906 11547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:22:09.508317 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 11480
I20260928 12:22:09.508405 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:22:09.508592 2272 external_mini_cluster.cc:1428] Setting key 61488357931ba56e790d2df03c4f39bc
I20260928 12:22:09.508816 11547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6d8c9dd92d049a19e3f02ce35a03894: Bootstrap starting.
I20260928 12:22:09.509545 11547 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6d8c9dd92d049a19e3f02ce35a03894: Neither blocks nor log segments found. Creating new log.
I20260928 12:22:09.509843 11547 log.cc:824] T 00000000000000000000000000000000 P b6d8c9dd92d049a19e3f02ce35a03894: Log is configured to *not* fsync() on all Append() calls
I20260928 12:22:09.510485 11547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6d8c9dd92d049a19e3f02ce35a03894: No bootstrap required, opened a new log
I20260928 12:22:09.512319 11547 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6d8c9dd92d049a19e3f02ce35a03894 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6d8c9dd92d049a19e3f02ce35a03894" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 41069 } }
I20260928 12:22:09.512463 11547 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6d8c9dd92d049a19e3f02ce35a03894 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:22:09.512485 11547 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6d8c9dd92d049a19e3f02ce35a03894 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6d8c9dd92d049a19e3f02ce35a03894, State: Initialized, Role: FOLLOWER
I20260928 12:22:09.512611 11547 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6d8c9dd92d049a19e3f02ce35a03894 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6d8c9dd92d049a19e3f02ce35a03894" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 41069 } }
I20260928 12:22:09.512694 11547 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6d8c9dd92d049a19e3f02ce35a03894 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:22:09.512720 11547 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6d8c9dd92d049a19e3f02ce35a03894 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:22:09.512768 11547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6d8c9dd92d049a19e3f02ce35a03894 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:22:09.513474 11547 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6d8c9dd92d049a19e3f02ce35a03894 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6d8c9dd92d049a19e3f02ce35a03894" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 41069 } }
I20260928 12:22:09.513584 11547 leader_election.cc:302] T 00000000000000000000000000000000 P b6d8c9dd92d049a19e3f02ce35a03894 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6d8c9dd92d049a19e3f02ce35a03894; no voters:
I20260928 12:22:09.513753 11547 leader_election.cc:288] T 00000000000000000000000000000000 P b6d8c9dd92d049a19e3f02ce35a03894 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:22:09.513794 11552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6d8c9dd92d049a19e3f02ce35a03894 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:22:09.513943 11552 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6d8c9dd92d049a19e3f02ce35a03894 [term 1 LEADER]: Becoming Leader. State: Replica: b6d8c9dd92d049a19e3f02ce35a03894, State: Running, Role: LEADER
I20260928 12:22:09.514073 11547 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6d8c9dd92d049a19e3f02ce35a03894 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:22:09.514077 11552 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6d8c9dd92d049a19e3f02ce35a03894 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6d8c9dd92d049a19e3f02ce35a03894" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 41069 } }
I20260928 12:22:09.514460 11550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:09.509127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60224 (local address 127.2.56.62:41069)
0928 12:22:09.509392 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:09.509402 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:09.509429 (+ 27us) server_negotiation.cc:407] Connection header received
0928 12:22:09.509934 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:09.509945 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:09.510019 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:09.510162 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:09.510395 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:09.511354 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:09.512175 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:09.512677 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:09.512816 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:09.513588 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:22:09.513593 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:22:09.513593 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:22:09.513841 (+ 248us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:22:09.513890 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:22:09.513909 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:22:09.514147 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0928 12:22:09.514187 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 12:22:09.514280 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":93,"threads_started":1}
I20260928 12:22:09.514698 11553 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6d8c9dd92d049a19e3f02ce35a03894 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6d8c9dd92d049a19e3f02ce35a03894" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6d8c9dd92d049a19e3f02ce35a03894" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 41069 } } }
I20260928 12:22:09.514780 11553 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6d8c9dd92d049a19e3f02ce35a03894 [sys.catalog]: This master's current role is: LEADER
I20260928 12:22:09.515172 11554 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6d8c9dd92d049a19e3f02ce35a03894 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6d8c9dd92d049a19e3f02ce35a03894. Latest consensus state: current_term: 1 leader_uuid: "b6d8c9dd92d049a19e3f02ce35a03894" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6d8c9dd92d049a19e3f02ce35a03894" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 41069 } } }
I20260928 12:22:09.515301 11554 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6d8c9dd92d049a19e3f02ce35a03894 [sys.catalog]: This master's current role is: LEADER
I20260928 12:22:09.515427 11561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:22:09.515929 11561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:22:09.517431 11561 catalog_manager.cc:1347] Generated new cluster ID: b5975e3237c04f788ed8e950b31bcc20
I20260928 12:22:09.517484 11561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:22:09.528445 11561 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:22:09.528949 11561 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:22:09.535601 11561 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6d8c9dd92d049a19e3f02ce35a03894: Generated new TSK 0
I20260928 12:22:09.535780 11561 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:22:09.544934 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.1:0
--local_ip_for_outbound_sockets=127.2.56.1
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:41069
--builtin_ntp_servers=127.2.56.20:34117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 12:22:09.632033 11571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:22:09.632279 11571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:22:09.632300 11571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:22:09.633527 11571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:22:09.633565 11571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:22:09.633590 11571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.1
I20260928 12:22:09.634956 11571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:34117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:41069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.1
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:22:09.635211 11571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:22:09.635527 11571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:22:09.641639 11576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:22:09.641666 11579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:22:09.641916 11577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:22:09.642377 11571 server_base.cc:1034] running on GCE node
I20260928 12:22:09.642616 11571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:22:09.642889 11571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:22:09.644052 11571 hybrid_clock.cc:648] HybridClock initialized: now 1790598129644037 us; error 33 us; skew 500 ppm
I20260928 12:22:09.645349 11571 webserver.cc:460] Webserver started at http://127.2.56.1:45113/ using document root <none> and password file <none>
I20260928 12:22:09.645598 11571 fs_manager.cc:359] Metadata directory not provided
I20260928 12:22:09.645640 11571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:22:09.645751 11571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:22:09.646927 11571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/data/instance:
uuid: "89841ed9ad834208b4107053451029a1"
format_stamp: "Formatted at 2026-09-28 12:22:09 on dist-test-slave-920p"
server_key: "1020cdbd7660c65962cdf0d5ca01ee4c"
server_key_iv: "defc94441e67526f123a41e56a2cef36"
server_key_version: "encryptionkey@0"
I20260928 12:22:09.647246 11571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance:
uuid: "89841ed9ad834208b4107053451029a1"
format_stamp: "Formatted at 2026-09-28 12:22:09 on dist-test-slave-920p"
server_key: "1020cdbd7660c65962cdf0d5ca01ee4c"
server_key_iv: "defc94441e67526f123a41e56a2cef36"
server_key_version: "encryptionkey@0"
I20260928 12:22:09.648505 11571 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:22:09.649168 11585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:09.649348 11571 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:22:09.649438 11571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/wal
uuid: "89841ed9ad834208b4107053451029a1"
format_stamp: "Formatted at 2026-09-28 12:22:09 on dist-test-slave-920p"
server_key: "1020cdbd7660c65962cdf0d5ca01ee4c"
server_key_iv: "defc94441e67526f123a41e56a2cef36"
server_key_version: "encryptionkey@0"
I20260928 12:22:09.649501 11571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:22:09.668217 11571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:22:09.668593 11571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:22:09.668710 11571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:22:09.668931 11571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:22:09.669317 11571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:22:09.669358 11571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:09.669387 11571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:22:09.669399 11571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:09.675621 11571 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.1:36049
I20260928 12:22:09.675691 11698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.1:36049 every 8 connection(s)
I20260928 12:22:09.676048 11571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
I20260928 12:22:09.680740 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 11571
I20260928 12:22:09.680826 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance
I20260928 12:22:09.681034 2272 external_mini_cluster.cc:1428] Setting key 3a0ae7975c4aec7348e7daffe02bc466
I20260928 12:22:09.682029 11550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:09.676826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:43249 (local address 127.2.56.62:41069)
0928 12:22:09.676904 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:09.676906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:09.677154 (+ 248us) server_negotiation.cc:407] Connection header received
0928 12:22:09.677811 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:09.677813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:09.677827 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:09.677890 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:09.678602 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:09.679150 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:09.680270 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:09.680498 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:09.680571 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:09.681673 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:22:09.681674 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:22:09.681675 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:22:09.681680 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:22:09.681702 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:22:09.681706 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:22:09.681782 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 12:22:09.681902 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 12:22:09.681939 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 12:22:09.682274 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.2:0
--local_ip_for_outbound_sockets=127.2.56.2
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:41069
--builtin_ntp_servers=127.2.56.20:34117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 12:22:09.682507 11701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:09.676826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:41069 (local address 127.2.56.1:43249)
0928 12:22:09.677026 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:09.677064 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:09.677704 (+ 640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:09.677964 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:09.677968 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:09.677977 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:09.678516 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:09.678521 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:09.679231 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:09.679232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:09.680088 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:09.680090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:09.680593 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:09.680594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:09.680728 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:09.681482 (+ 754us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:22:09.681482 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:09.681528 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:22:09.681562 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:22:09.681785 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:22:09.681793 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:22:09.681798 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:22:09.682133 (+ 335us) client_negotiation.cc:240] Negotiation successful
0928 12:22:09.682254 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":74,"threads_started":1}
I20260928 12:22:09.684068 11699 heartbeater.cc:347] Connected to a master server at 127.2.56.62:41069
I20260928 12:22:09.684168 11699 heartbeater.cc:464] Registering TS with master...
I20260928 12:22:09.684386 11699 heartbeater.cc:511] Master 127.2.56.62:41069 requested a full tablet report, sending...
I20260928 12:22:09.684993 11511 ts_manager.cc:194] Registered new tserver with Master: 89841ed9ad834208b4107053451029a1 (127.2.56.1:36049)
I20260928 12:22:09.685714 11511 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.56.1:43249
W20260928 12:22:09.771878 11702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:22:09.772111 11702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:22:09.772133 11702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:22:09.773370 11702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:22:09.773414 11702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:22:09.773442 11702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.2
I20260928 12:22:09.774792 11702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:34117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:41069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.2
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:22:09.775041 11702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:22:09.775370 11702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:22:09.780633 11708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:22:09.780661 11710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:22:09.780642 11707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:22:09.780659 11702 server_base.cc:1034] running on GCE node
I20260928 12:22:09.781078 11702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:22:09.781361 11702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:22:09.782511 11702 hybrid_clock.cc:648] HybridClock initialized: now 1790598129782495 us; error 34 us; skew 500 ppm
I20260928 12:22:09.783854 11702 webserver.cc:460] Webserver started at http://127.2.56.2:33923/ using document root <none> and password file <none>
I20260928 12:22:09.784093 11702 fs_manager.cc:359] Metadata directory not provided
I20260928 12:22:09.784140 11702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:22:09.784272 11702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:22:09.785475 11702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/data/instance:
uuid: "0a2aa1bd3b5e49829bb520666009ff41"
format_stamp: "Formatted at 2026-09-28 12:22:09 on dist-test-slave-920p"
server_key: "df44e64290a0491db44dafc4e1fd4865"
server_key_iv: "4db81cf41273591018f2917d0f4aa1dc"
server_key_version: "encryptionkey@0"
I20260928 12:22:09.785836 11702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance:
uuid: "0a2aa1bd3b5e49829bb520666009ff41"
format_stamp: "Formatted at 2026-09-28 12:22:09 on dist-test-slave-920p"
server_key: "df44e64290a0491db44dafc4e1fd4865"
server_key_iv: "4db81cf41273591018f2917d0f4aa1dc"
server_key_version: "encryptionkey@0"
I20260928 12:22:09.787122 11702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:22:09.787807 11716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:09.787938 11702 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:22:09.788017 11702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/wal
uuid: "0a2aa1bd3b5e49829bb520666009ff41"
format_stamp: "Formatted at 2026-09-28 12:22:09 on dist-test-slave-920p"
server_key: "df44e64290a0491db44dafc4e1fd4865"
server_key_iv: "4db81cf41273591018f2917d0f4aa1dc"
server_key_version: "encryptionkey@0"
I20260928 12:22:09.788100 11702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:22:09.804132 11702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:22:09.804471 11702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:22:09.804605 11702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:22:09.804845 11702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:22:09.805212 11702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:22:09.805246 11702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:09.805339 11702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:22:09.805368 11702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:09.811722 11702 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.2:42931
I20260928 12:22:09.811792 11829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.2:42931 every 8 connection(s)
I20260928 12:22:09.812121 11702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
I20260928 12:22:09.817749 11550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:09.812881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:55943 (local address 127.2.56.62:41069)
0928 12:22:09.812948 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:09.812950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:09.813206 (+ 256us) server_negotiation.cc:407] Connection header received
0928 12:22:09.813823 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:09.813825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:09.813849 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:09.813904 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:09.814662 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:09.815317 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:09.816336 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:09.816549 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:09.816605 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:09.817439 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:22:09.817440 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:22:09.817441 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:22:09.817446 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:22:09.817466 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:22:09.817470 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:22:09.817515 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 12:22:09.817594 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 12:22:09.817626 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 12:22:09.817947 11832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:09.812868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:41069 (local address 127.2.56.2:55943)
0928 12:22:09.813067 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:09.813106 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:09.813712 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:09.813984 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:09.813987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:09.813998 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:09.814576 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:09.814582 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:09.815392 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:09.815394 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:09.816251 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:09.816254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:09.816615 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:09.816617 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:09.816760 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:09.817301 (+ 541us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:22:09.817301 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:09.817333 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:22:09.817359 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:22:09.817521 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:22:09.817526 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:22:09.817529 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:22:09.817592 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 12:22:09.817709 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":88,"threads_started":1}
I20260928 12:22:09.817996 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 11702
I20260928 12:22:09.818084 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance
I20260928 12:22:09.818370 2272 external_mini_cluster.cc:1428] Setting key f56ecc68ba8a63379e6785eecbd7624f
I20260928 12:22:09.818642 11830 heartbeater.cc:347] Connected to a master server at 127.2.56.62:41069
I20260928 12:22:09.818751 11830 heartbeater.cc:464] Registering TS with master...
I20260928 12:22:09.818977 11830 heartbeater.cc:511] Master 127.2.56.62:41069 requested a full tablet report, sending...
I20260928 12:22:09.819396 11511 ts_manager.cc:194] Registered new tserver with Master: 0a2aa1bd3b5e49829bb520666009ff41 (127.2.56.2:42931)
I20260928 12:22:09.819481 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.3:0
--local_ip_for_outbound_sockets=127.2.56.3
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:41069
--builtin_ntp_servers=127.2.56.20:34117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 12:22:09.819866 11511 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.56.2:55943
W20260928 12:22:09.909976 11833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:22:09.910238 11833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:22:09.910260 11833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:22:09.911501 11833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:22:09.911552 11833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:22:09.911578 11833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.3
I20260928 12:22:09.912992 11833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:34117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:41069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.3
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:22:09.913260 11833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:22:09.913668 11833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:22:09.919047 11838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:22:09.919044 11841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:22:09.919093 11839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:22:09.919405 11833 server_base.cc:1034] running on GCE node
I20260928 12:22:09.919595 11833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:22:09.919839 11833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:22:09.920994 11833 hybrid_clock.cc:648] HybridClock initialized: now 1790598129920977 us; error 35 us; skew 500 ppm
I20260928 12:22:09.922384 11833 webserver.cc:460] Webserver started at http://127.2.56.3:42219/ using document root <none> and password file <none>
I20260928 12:22:09.922614 11833 fs_manager.cc:359] Metadata directory not provided
I20260928 12:22:09.922659 11833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:22:09.922828 11833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:22:09.924074 11833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/data/instance:
uuid: "1802a8c0be974ffcad950148cab23930"
format_stamp: "Formatted at 2026-09-28 12:22:09 on dist-test-slave-920p"
server_key: "e88c407ed03098e40e77e02d9096a83f"
server_key_iv: "3b3c673f1f59170cc2d053b01e6149d3"
server_key_version: "encryptionkey@0"
I20260928 12:22:09.924448 11833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance:
uuid: "1802a8c0be974ffcad950148cab23930"
format_stamp: "Formatted at 2026-09-28 12:22:09 on dist-test-slave-920p"
server_key: "e88c407ed03098e40e77e02d9096a83f"
server_key_iv: "3b3c673f1f59170cc2d053b01e6149d3"
server_key_version: "encryptionkey@0"
I20260928 12:22:09.925979 11833 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:22:09.926798 11847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:09.926927 11833 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:22:09.927012 11833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/wal
uuid: "1802a8c0be974ffcad950148cab23930"
format_stamp: "Formatted at 2026-09-28 12:22:09 on dist-test-slave-920p"
server_key: "e88c407ed03098e40e77e02d9096a83f"
server_key_iv: "3b3c673f1f59170cc2d053b01e6149d3"
server_key_version: "encryptionkey@0"
I20260928 12:22:09.927085 11833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:22:09.940495 11833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:22:09.940860 11833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:22:09.941000 11833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:22:09.941248 11833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:22:09.941660 11833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:22:09.941696 11833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:09.941720 11833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:22:09.941743 11833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:09.948285 11833 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.3:46121
I20260928 12:22:09.948343 11960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.3:46121 every 8 connection(s)
I20260928 12:22:09.948705 11833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
I20260928 12:22:09.954694 11550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:09.949615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:43719 (local address 127.2.56.62:41069)
0928 12:22:09.949683 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:09.949686 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:09.949981 (+ 295us) server_negotiation.cc:407] Connection header received
0928 12:22:09.950604 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:09.950605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:09.950621 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:09.950678 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:09.951434 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:09.952029 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:09.953166 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:09.953443 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:09.953502 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:09.954405 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:22:09.954407 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:22:09.954407 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:22:09.954413 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:22:09.954433 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:22:09.954437 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:22:09.954483 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 12:22:09.954566 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 12:22:09.954599 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 12:22:09.954850 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 11833
I20260928 12:22:09.954944 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance
I20260928 12:22:09.954937 11963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:09.949613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:41069 (local address 127.2.56.3:43719)
0928 12:22:09.949842 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:09.949883 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:09.950489 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:09.950765 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:09.950769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:09.950779 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:09.951340 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:09.951345 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:09.952119 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:09.952120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:09.953085 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:09.953087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:09.953524 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:09.953525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:09.953667 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:09.954256 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:22:09.954256 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:09.954291 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:22:09.954320 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:22:09.954488 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:22:09.954494 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:22:09.954497 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:22:09.954565 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 12:22:09.954685 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":83,"threads_started":1}
I20260928 12:22:09.955150 2272 external_mini_cluster.cc:1428] Setting key c2a66a54fa1ab2ce245dca07babc8215
I20260928 12:22:09.955416 11961 heartbeater.cc:347] Connected to a master server at 127.2.56.62:41069
I20260928 12:22:09.955516 11961 heartbeater.cc:464] Registering TS with master...
I20260928 12:22:09.955719 11961 heartbeater.cc:511] Master 127.2.56.62:41069 requested a full tablet report, sending...
I20260928 12:22:09.956105 11511 ts_manager.cc:194] Registered new tserver with Master: 1802a8c0be974ffcad950148cab23930 (127.2.56.3:46121)
I20260928 12:22:09.956488 11511 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.56.3:43719
I20260928 12:22:09.967187 2272 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 12:22:09.969399 11550 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:22:09.968777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60230 (local address 127.2.56.62:41069)
0928 12:22:09.968855 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:09.968858 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:09.968964 (+ 106us) server_negotiation.cc:407] Connection header received
0928 12:22:09.969038 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:09.969039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:09.969052 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:09.969103 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:09.969299 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60230: BlockingRecv error: recv got EOF from 127.0.0.1:60230 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 12:22:09.969473 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 11571
I20260928 12:22:09.974251 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 11702
I20260928 12:22:09.978690 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 11833
I20260928 12:22:09.982839 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 11480
2026-09-28T12:22:09Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (625 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T12:22:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:22:10Z Disabled control of system clock
I20260928 12:22:10.010205 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:35761
--webserver_interface=127.2.56.62
--webserver_port=0
--builtin_ntp_servers=127.2.56.20:42735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:35761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 12:22:10.096580 11972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:22:10.096895 11972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:22:10.096921 11972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:22:10.098323 11972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:22:10.098376 11972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:22:10.098387 11972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:22:10.098398 11972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:22:10.098413 11972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:22:10.099928 11972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:42735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:35761
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:35761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:22:10.100224 11972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:22:10.100562 11972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:22:10.106194 11978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:22:10.106195 11977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:22:10.106194 11980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:22:10.106523 11972 server_base.cc:1034] running on GCE node
I20260928 12:22:10.106817 11972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:22:10.107162 11972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:22:10.108309 11972 hybrid_clock.cc:648] HybridClock initialized: now 1790598130108301 us; error 43 us; skew 500 ppm
I20260928 12:22:10.109910 11972 webserver.cc:460] Webserver started at http://127.2.56.62:36319/ using document root <none> and password file <none>
I20260928 12:22:10.110181 11972 fs_manager.cc:359] Metadata directory not provided
I20260928 12:22:10.110231 11972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:22:10.110365 11972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:22:10.112056 11972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "50bcd631f12746e3810320495512f04d"
format_stamp: "Formatted at 2026-09-28 12:22:10 on dist-test-slave-920p"
server_key: "d8af0f91e4b9f81bc416a0c9e498afec"
server_key_iv: "fe6bda8129b9c6bca5c23b6268eb7499"
server_key_version: "encryptionkey@0"
I20260928 12:22:10.112528 11972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "50bcd631f12746e3810320495512f04d"
format_stamp: "Formatted at 2026-09-28 12:22:10 on dist-test-slave-920p"
server_key: "d8af0f91e4b9f81bc416a0c9e498afec"
server_key_iv: "fe6bda8129b9c6bca5c23b6268eb7499"
server_key_version: "encryptionkey@0"
I20260928 12:22:10.114136 11972 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260928 12:22:10.115104 11986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:10.115305 11972 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:22:10.115404 11972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "50bcd631f12746e3810320495512f04d"
format_stamp: "Formatted at 2026-09-28 12:22:10 on dist-test-slave-920p"
server_key: "d8af0f91e4b9f81bc416a0c9e498afec"
server_key_iv: "fe6bda8129b9c6bca5c23b6268eb7499"
server_key_version: "encryptionkey@0"
I20260928 12:22:10.115492 11972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:22:10.119714 11972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:22:10.120097 11972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:22:10.120244 11972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:22:10.125232 11972 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:35761
I20260928 12:22:10.125257 12038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:35761 every 8 connection(s)
I20260928 12:22:10.125775 11972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:22:10.126647 12039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:22:10.130157 12039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50bcd631f12746e3810320495512f04d: Bootstrap starting.
I20260928 12:22:10.130895 12039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50bcd631f12746e3810320495512f04d: Neither blocks nor log segments found. Creating new log.
I20260928 12:22:10.131253 12039 log.cc:824] T 00000000000000000000000000000000 P 50bcd631f12746e3810320495512f04d: Log is configured to *not* fsync() on all Append() calls
I20260928 12:22:10.132071 12039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50bcd631f12746e3810320495512f04d: No bootstrap required, opened a new log
I20260928 12:22:10.134280 12039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50bcd631f12746e3810320495512f04d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50bcd631f12746e3810320495512f04d" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 35761 } }
I20260928 12:22:10.134481 12039 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50bcd631f12746e3810320495512f04d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:22:10.134521 12039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50bcd631f12746e3810320495512f04d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50bcd631f12746e3810320495512f04d, State: Initialized, Role: FOLLOWER
I20260928 12:22:10.134698 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 11972
I20260928 12:22:10.134718 12039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50bcd631f12746e3810320495512f04d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50bcd631f12746e3810320495512f04d" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 35761 } }
I20260928 12:22:10.134790 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:22:10.134819 12039 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50bcd631f12746e3810320495512f04d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:22:10.134860 12039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50bcd631f12746e3810320495512f04d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:22:10.134914 12039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50bcd631f12746e3810320495512f04d [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:22:10.134989 2272 external_mini_cluster.cc:1428] Setting key f28525bbce93d231ee3c8ae3ceb285c6
I20260928 12:22:10.135654 12039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50bcd631f12746e3810320495512f04d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50bcd631f12746e3810320495512f04d" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 35761 } }
I20260928 12:22:10.135782 12039 leader_election.cc:302] T 00000000000000000000000000000000 P 50bcd631f12746e3810320495512f04d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50bcd631f12746e3810320495512f04d; no voters:
I20260928 12:22:10.135947 12039 leader_election.cc:288] T 00000000000000000000000000000000 P 50bcd631f12746e3810320495512f04d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:22:10.136010 12044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50bcd631f12746e3810320495512f04d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:22:10.136171 12044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50bcd631f12746e3810320495512f04d [term 1 LEADER]: Becoming Leader. State: Replica: 50bcd631f12746e3810320495512f04d, State: Running, Role: LEADER
I20260928 12:22:10.136243 12039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50bcd631f12746e3810320495512f04d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:22:10.136360 12044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50bcd631f12746e3810320495512f04d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50bcd631f12746e3810320495512f04d" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 35761 } }
I20260928 12:22:10.136775 12044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50bcd631f12746e3810320495512f04d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50bcd631f12746e3810320495512f04d. Latest consensus state: current_term: 1 leader_uuid: "50bcd631f12746e3810320495512f04d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50bcd631f12746e3810320495512f04d" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 35761 } } }
I20260928 12:22:10.136855 12044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50bcd631f12746e3810320495512f04d [sys.catalog]: This master's current role is: LEADER
I20260928 12:22:10.137174 12048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:22:10.137349 12045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50bcd631f12746e3810320495512f04d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50bcd631f12746e3810320495512f04d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50bcd631f12746e3810320495512f04d" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 35761 } } }
I20260928 12:22:10.137424 12045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50bcd631f12746e3810320495512f04d [sys.catalog]: This master's current role is: LEADER
I20260928 12:22:10.137872 12043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:10.135545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52394 (local address 127.2.56.62:35761)
0928 12:22:10.135819 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:10.135830 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:10.135869 (+ 39us) server_negotiation.cc:407] Connection header received
0928 12:22:10.136020 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:10.136029 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:10.136088 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:10.136559 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:10.137147 (+ 588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:22:10.137151 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:22:10.137151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:22:10.137154 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:22:10.137195 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:22:10.137211 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:22:10.137533 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0928 12:22:10.137570 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 12:22:10.137646 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":99,"threads_started":1}
I20260928 12:22:10.138075 12048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:22:10.139845 12048 catalog_manager.cc:1347] Generated new cluster ID: 1e9ba850da9f43279378bb0c3cfe2baf
I20260928 12:22:10.139904 12048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:22:10.157624 12048 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:22:10.157825 12048 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:22:10.161377 12048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50bcd631f12746e3810320495512f04d: Generated new TSK 0
I20260928 12:22:10.161597 12048 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:22:10.163571 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.1:0
--local_ip_for_outbound_sockets=127.2.56.1
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:35761
--builtin_ntp_servers=127.2.56.20:42735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 12:22:10.251003 12063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:22:10.251240 12063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:22:10.251273 12063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:22:10.252493 12063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:22:10.252529 12063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:22:10.252562 12063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.1
I20260928 12:22:10.253943 12063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:42735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:35761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.1
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:22:10.254187 12063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:22:10.254485 12063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:22:10.260608 12063 server_base.cc:1034] running on GCE node
W20260928 12:22:10.260555 12068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:22:10.260526 12071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:22:10.260717 12069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:22:10.261056 12063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:22:10.261358 12063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:22:10.262522 12063 hybrid_clock.cc:648] HybridClock initialized: now 1790598130262507 us; error 33 us; skew 500 ppm
I20260928 12:22:10.263875 12063 webserver.cc:460] Webserver started at http://127.2.56.1:35293/ using document root <none> and password file <none>
I20260928 12:22:10.264130 12063 fs_manager.cc:359] Metadata directory not provided
I20260928 12:22:10.264176 12063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:22:10.264307 12063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:22:10.265550 12063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/data/instance:
uuid: "b70731feea8246999044d15c6499de88"
format_stamp: "Formatted at 2026-09-28 12:22:10 on dist-test-slave-920p"
server_key: "30d96a35f65911276366c9d64dbb9cab"
server_key_iv: "8fce0b5a8d857be10dec949c98967a02"
server_key_version: "encryptionkey@0"
I20260928 12:22:10.265906 12063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance:
uuid: "b70731feea8246999044d15c6499de88"
format_stamp: "Formatted at 2026-09-28 12:22:10 on dist-test-slave-920p"
server_key: "30d96a35f65911276366c9d64dbb9cab"
server_key_iv: "8fce0b5a8d857be10dec949c98967a02"
server_key_version: "encryptionkey@0"
I20260928 12:22:10.267237 12063 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:22:10.267992 12077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:10.268172 12063 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:22:10.268267 12063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/wal
uuid: "b70731feea8246999044d15c6499de88"
format_stamp: "Formatted at 2026-09-28 12:22:10 on dist-test-slave-920p"
server_key: "30d96a35f65911276366c9d64dbb9cab"
server_key_iv: "8fce0b5a8d857be10dec949c98967a02"
server_key_version: "encryptionkey@0"
I20260928 12:22:10.268324 12063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:22:10.271899 12063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:22:10.272157 12063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:22:10.272261 12063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:22:10.272445 12063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:22:10.272717 12063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:22:10.272753 12063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:10.272776 12063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:22:10.272791 12063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:10.279006 12063 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.1:39481
I20260928 12:22:10.279062 12190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.1:39481 every 8 connection(s)
I20260928 12:22:10.279399 12063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
I20260928 12:22:10.282447 12043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:10.280164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:59147 (local address 127.2.56.62:35761)
0928 12:22:10.280240 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:10.280243 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:10.280428 (+ 185us) server_negotiation.cc:407] Connection header received
0928 12:22:10.280983 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:10.280984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:10.280993 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:10.281042 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:10.282216 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:22:10.282218 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:22:10.282219 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:22:10.282223 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:22:10.282241 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:22:10.282246 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:22:10.282294 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 12:22:10.282359 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 12:22:10.282374 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 12:22:10.282616 12193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:10.280106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:35761 (local address 127.2.56.1:59147)
0928 12:22:10.280299 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:10.280338 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:10.280892 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:10.281109 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:10.281112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:10.281122 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:10.282056 (+ 934us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:22:10.282057 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:10.282087 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:22:10.282120 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:22:10.282297 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:22:10.282300 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:22:10.282303 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:22:10.282359 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 12:22:10.282416 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260928 12:22:10.283113 12191 heartbeater.cc:347] Connected to a master server at 127.2.56.62:35761
I20260928 12:22:10.283198 12191 heartbeater.cc:464] Registering TS with master...
I20260928 12:22:10.283324 12191 heartbeater.cc:511] Master 127.2.56.62:35761 requested a full tablet report, sending...
I20260928 12:22:10.283866 12003 ts_manager.cc:194] Registered new tserver with Master: b70731feea8246999044d15c6499de88 (127.2.56.1:39481)
I20260928 12:22:10.289311 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 12063
I20260928 12:22:10.289408 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance
I20260928 12:22:10.289610 2272 external_mini_cluster.cc:1428] Setting key 1af3401fdc733b0d494ce3fc6791b681
I20260928 12:22:10.290505 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.2:0
--local_ip_for_outbound_sockets=127.2.56.2
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:35761
--builtin_ntp_servers=127.2.56.20:42735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 12:22:10.376545 12194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:22:10.376778 12194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:22:10.376799 12194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:22:10.378000 12194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:22:10.378049 12194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:22:10.378074 12194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.2
I20260928 12:22:10.379387 12194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:42735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:35761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.2
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:22:10.379593 12194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:22:10.379863 12194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:22:10.384910 12200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:22:10.384953 12199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:22:10.385074 12194 server_base.cc:1034] running on GCE node
W20260928 12:22:10.385145 12202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:22:10.385392 12194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:22:10.385646 12194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:22:10.386808 12194 hybrid_clock.cc:648] HybridClock initialized: now 1790598130386795 us; error 30 us; skew 500 ppm
I20260928 12:22:10.388154 12194 webserver.cc:460] Webserver started at http://127.2.56.2:41487/ using document root <none> and password file <none>
I20260928 12:22:10.388386 12194 fs_manager.cc:359] Metadata directory not provided
I20260928 12:22:10.388433 12194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:22:10.388567 12194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:22:10.389803 12194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/data/instance:
uuid: "0dcfe9ae77014b0dab4096ebfaf5cc28"
format_stamp: "Formatted at 2026-09-28 12:22:10 on dist-test-slave-920p"
server_key: "66b130a81c5b37d5dbd0879751aea330"
server_key_iv: "8b997c804d970ced82d1c48957948a41"
server_key_version: "encryptionkey@0"
I20260928 12:22:10.390154 12194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance:
uuid: "0dcfe9ae77014b0dab4096ebfaf5cc28"
format_stamp: "Formatted at 2026-09-28 12:22:10 on dist-test-slave-920p"
server_key: "66b130a81c5b37d5dbd0879751aea330"
server_key_iv: "8b997c804d970ced82d1c48957948a41"
server_key_version: "encryptionkey@0"
I20260928 12:22:10.391520 12194 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:22:10.392295 12208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:10.392465 12194 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:22:10.392553 12194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/wal
uuid: "0dcfe9ae77014b0dab4096ebfaf5cc28"
format_stamp: "Formatted at 2026-09-28 12:22:10 on dist-test-slave-920p"
server_key: "66b130a81c5b37d5dbd0879751aea330"
server_key_iv: "8b997c804d970ced82d1c48957948a41"
server_key_version: "encryptionkey@0"
I20260928 12:22:10.392616 12194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:22:10.396153 12194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:22:10.396417 12194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:22:10.396533 12194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:22:10.396723 12194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:22:10.397027 12194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:22:10.397058 12194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:10.397084 12194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:22:10.397107 12194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:10.403326 12194 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.2:39407
I20260928 12:22:10.403365 12321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.2:39407 every 8 connection(s)
I20260928 12:22:10.403702 12194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
I20260928 12:22:10.405366 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 12194
I20260928 12:22:10.405479 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance
I20260928 12:22:10.405663 2272 external_mini_cluster.cc:1428] Setting key 4c9b1a8236711dfff1faadbd7b84891a
I20260928 12:22:10.406637 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.3:0
--local_ip_for_outbound_sockets=127.2.56.3
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:35761
--builtin_ntp_servers=127.2.56.20:42735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 12:22:10.407002 12043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:10.404457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:44881 (local address 127.2.56.62:35761)
0928 12:22:10.404524 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:10.404526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:10.404758 (+ 232us) server_negotiation.cc:407] Connection header received
0928 12:22:10.405301 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:10.405302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:10.405311 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:10.405396 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:10.406742 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:22:10.406744 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:22:10.406744 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:22:10.406749 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:22:10.406776 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:22:10.406781 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:22:10.406828 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 12:22:10.406909 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 12:22:10.406926 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 12:22:10.407207 12324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:10.404446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:35761 (local address 127.2.56.2:44881)
0928 12:22:10.404627 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:10.404673 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:10.405188 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:10.405476 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:10.405481 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:10.405493 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:10.406589 (+ 1096us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:22:10.406590 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:10.406624 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:22:10.406655 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:22:10.406830 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:22:10.406835 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:22:10.406839 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:22:10.406919 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 12:22:10.406987 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
I20260928 12:22:10.407653 12322 heartbeater.cc:347] Connected to a master server at 127.2.56.62:35761
I20260928 12:22:10.407739 12322 heartbeater.cc:464] Registering TS with master...
I20260928 12:22:10.407895 12322 heartbeater.cc:511] Master 127.2.56.62:35761 requested a full tablet report, sending...
I20260928 12:22:10.408303 12003 ts_manager.cc:194] Registered new tserver with Master: 0dcfe9ae77014b0dab4096ebfaf5cc28 (127.2.56.2:39407)
W20260928 12:22:10.493076 12325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:22:10.493330 12325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:22:10.493350 12325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:22:10.494544 12325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:22:10.494582 12325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:22:10.494608 12325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.3
I20260928 12:22:10.495980 12325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:42735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:35761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.3
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:22:10.496228 12325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:22:10.496537 12325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:22:10.501691 12330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:22:10.501742 12325 server_base.cc:1034] running on GCE node
W20260928 12:22:10.501710 12331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:22:10.501716 12333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:22:10.502015 12325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:22:10.502251 12325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:22:10.503429 12325 hybrid_clock.cc:648] HybridClock initialized: now 1790598130503414 us; error 33 us; skew 500 ppm
I20260928 12:22:10.504669 12325 webserver.cc:460] Webserver started at http://127.2.56.3:39643/ using document root <none> and password file <none>
I20260928 12:22:10.504876 12325 fs_manager.cc:359] Metadata directory not provided
I20260928 12:22:10.504917 12325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:22:10.505018 12325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:22:10.506183 12325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/data/instance:
uuid: "4859d7887c2e410abb43f1ea83ac02a3"
format_stamp: "Formatted at 2026-09-28 12:22:10 on dist-test-slave-920p"
server_key: "f76294e0d5bc04b9c7f8a5a7ac1a9da9"
server_key_iv: "8e8e27a9c6110914ca8cc9ccdf7b01c0"
server_key_version: "encryptionkey@0"
I20260928 12:22:10.506523 12325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance:
uuid: "4859d7887c2e410abb43f1ea83ac02a3"
format_stamp: "Formatted at 2026-09-28 12:22:10 on dist-test-slave-920p"
server_key: "f76294e0d5bc04b9c7f8a5a7ac1a9da9"
server_key_iv: "8e8e27a9c6110914ca8cc9ccdf7b01c0"
server_key_version: "encryptionkey@0"
I20260928 12:22:10.507835 12325 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 12:22:10.508455 12339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:10.508606 12325 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:22:10.508685 12325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/wal
uuid: "4859d7887c2e410abb43f1ea83ac02a3"
format_stamp: "Formatted at 2026-09-28 12:22:10 on dist-test-slave-920p"
server_key: "f76294e0d5bc04b9c7f8a5a7ac1a9da9"
server_key_iv: "8e8e27a9c6110914ca8cc9ccdf7b01c0"
server_key_version: "encryptionkey@0"
I20260928 12:22:10.508742 12325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:22:10.512197 12325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:22:10.512440 12325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:22:10.512532 12325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:22:10.512705 12325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:22:10.512964 12325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:22:10.512991 12325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:10.513015 12325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:22:10.513031 12325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:10.519099 12325 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.3:38219
I20260928 12:22:10.519164 12452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.3:38219 every 8 connection(s)
I20260928 12:22:10.519459 12325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
I20260928 12:22:10.521724 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 12325
I20260928 12:22:10.521795 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance
I20260928 12:22:10.521981 2272 external_mini_cluster.cc:1428] Setting key dd48becaff962e93edd28f8d8630b783
I20260928 12:22:10.522554 12043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:10.520158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:33935 (local address 127.2.56.62:35761)
0928 12:22:10.520233 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:10.520237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:10.520465 (+ 228us) server_negotiation.cc:407] Connection header received
0928 12:22:10.520959 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:10.520961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:10.520967 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:10.521016 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:10.522240 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:22:10.522242 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:22:10.522242 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:22:10.522247 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:22:10.522265 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:22:10.522270 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:22:10.522322 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 12:22:10.522459 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 12:22:10.522477 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 12:22:10.522704 12455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:10.520148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:35761 (local address 127.2.56.3:33935)
0928 12:22:10.520344 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:10.520379 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:10.520874 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:10.521079 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:10.521083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:10.521092 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:10.522056 (+ 964us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:22:10.522056 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:10.522096 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:22:10.522134 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:22:10.522322 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:22:10.522329 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:22:10.522333 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:22:10.522426 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 12:22:10.522492 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":98,"threads_started":1}
I20260928 12:22:10.523049 12453 heartbeater.cc:347] Connected to a master server at 127.2.56.62:35761
I20260928 12:22:10.523141 12453 heartbeater.cc:464] Registering TS with master...
I20260928 12:22:10.523290 12453 heartbeater.cc:511] Master 127.2.56.62:35761 requested a full tablet report, sending...
I20260928 12:22:10.523640 12003 ts_manager.cc:194] Registered new tserver with Master: 4859d7887c2e410abb43f1ea83ac02a3 (127.2.56.3:38219)
I20260928 12:22:10.533674 2272 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 12:22:10.535539 12043 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:22:10.535099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52402 (local address 127.2.56.62:35761)
0928 12:22:10.535144 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:10.535146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:10.535265 (+ 119us) server_negotiation.cc:407] Connection header received
0928 12:22:10.535307 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:10.535307 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:10.535312 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:10.535354 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:10.535468 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52402: BlockingRecv error: recv got EOF from 127.0.0.1:52402 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:22:10.535573 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 12063
I20260928 12:22:10.540207 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 12194
I20260928 12:22:10.544409 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 12325
I20260928 12:22:10.548172 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 11972
2026-09-28T12:22:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (565 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:22:10 dist-test-slave-920p krb5kdc[12463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:22:10 dist-test-slave-920p krb5kdc[12463](info): set up 2 sockets
Sep 28 12:22:10 dist-test-slave-920p krb5kdc[12463](info): commencing operation
krb5kdc: starting...
W20260928 12:22:11.514302 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:22:11 dist-test-slave-920p krb5kdc[12463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:22:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:22:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
I20260928 12:22:11.631897 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:42677
--webserver_interface=127.2.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:34913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:42677
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:22:11.719241 12476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:22:11.719493 12476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:22:11.719527 12476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:22:11.720716 12476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:22:11.720755 12476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:22:11.720773 12476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:22:11.720789 12476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:22:11.720804 12476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:22:11.722362 12476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:34913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:42677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:42677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:22:11.722641 12476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:22:11.722862 12476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:22:11.728263 12482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:22:11.728261 12484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:22:11.728271 12481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:22:11.728511 12476 server_base.cc:1034] running on GCE node
I20260928 12:22:11.728798 12476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:22:11.729117 12476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:22:11.730250 12476 hybrid_clock.cc:648] HybridClock initialized: now 1790598131730240 us; error 30 us; skew 500 ppm
Sep 28 12:22:11 dist-test-slave-920p krb5kdc[12463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598131, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:22:11.732910 12476 init.cc:377] Logged in from keytab as kudu/127.2.56.62@KRBTEST.COM (short username kudu)
I20260928 12:22:11.733404 12476 webserver.cc:460] Webserver started at http://127.2.56.62:34285/ using document root <none> and password file <none>
I20260928 12:22:11.733635 12476 fs_manager.cc:359] Metadata directory not provided
I20260928 12:22:11.733685 12476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:22:11.733812 12476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:22:11.734972 12476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "acd98ef851974963b6577e07bc163181"
format_stamp: "Formatted at 2026-09-28 12:22:11 on dist-test-slave-920p"
server_key: "e90fbd8c379abb820ba3f8a6874e43b7"
server_key_iv: "43c08d87878453c3dcd8c8e9bf44f0f0"
server_key_version: "encryptionkey@0"
I20260928 12:22:11.735332 12476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "acd98ef851974963b6577e07bc163181"
format_stamp: "Formatted at 2026-09-28 12:22:11 on dist-test-slave-920p"
server_key: "e90fbd8c379abb820ba3f8a6874e43b7"
server_key_iv: "43c08d87878453c3dcd8c8e9bf44f0f0"
server_key_version: "encryptionkey@0"
I20260928 12:22:11.736712 12476 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:22:11.737473 12491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:11.737617 12476 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:22:11.737705 12476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "acd98ef851974963b6577e07bc163181"
format_stamp: "Formatted at 2026-09-28 12:22:11 on dist-test-slave-920p"
server_key: "e90fbd8c379abb820ba3f8a6874e43b7"
server_key_iv: "43c08d87878453c3dcd8c8e9bf44f0f0"
server_key_version: "encryptionkey@0"
I20260928 12:22:11.737771 12476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:22:11.750953 12476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:22:11.753153 12476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:22:11.753360 12476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:22:11.757827 12476 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:42677
I20260928 12:22:11.757874 12543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:42677 every 8 connection(s)
I20260928 12:22:11.758265 12476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:22:11.759378 12544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:22:11.762250 12544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acd98ef851974963b6577e07bc163181: Bootstrap starting.
I20260928 12:22:11.762953 12544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acd98ef851974963b6577e07bc163181: Neither blocks nor log segments found. Creating new log.
I20260928 12:22:11.763250 12544 log.cc:824] T 00000000000000000000000000000000 P acd98ef851974963b6577e07bc163181: Log is configured to *not* fsync() on all Append() calls
I20260928 12:22:11.763980 12544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acd98ef851974963b6577e07bc163181: No bootstrap required, opened a new log
I20260928 12:22:11.765805 12544 raft_consensus.cc:348] T 00000000000000000000000000000000 P acd98ef851974963b6577e07bc163181 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acd98ef851974963b6577e07bc163181" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 42677 } }
I20260928 12:22:11.765976 12544 raft_consensus.cc:374] T 00000000000000000000000000000000 P acd98ef851974963b6577e07bc163181 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:22:11.766011 12544 raft_consensus.cc:729] T 00000000000000000000000000000000 P acd98ef851974963b6577e07bc163181 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acd98ef851974963b6577e07bc163181, State: Initialized, Role: FOLLOWER
I20260928 12:22:11.766184 12544 consensus_queue.cc:260] T 00000000000000000000000000000000 P acd98ef851974963b6577e07bc163181 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acd98ef851974963b6577e07bc163181" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 42677 } }
I20260928 12:22:11.766249 12544 raft_consensus.cc:388] T 00000000000000000000000000000000 P acd98ef851974963b6577e07bc163181 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:22:11.766284 12544 raft_consensus.cc:482] T 00000000000000000000000000000000 P acd98ef851974963b6577e07bc163181 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:22:11.766335 12544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acd98ef851974963b6577e07bc163181 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:22:11.767019 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 12476
I20260928 12:22:11.767127 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:22:11.767333 2272 external_mini_cluster.cc:1428] Setting key c32597a61db091a82189d28cad64699d
I20260928 12:22:11.767426 12544 raft_consensus.cc:504] T 00000000000000000000000000000000 P acd98ef851974963b6577e07bc163181 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acd98ef851974963b6577e07bc163181" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 42677 } }
I20260928 12:22:11.767544 12544 leader_election.cc:302] T 00000000000000000000000000000000 P acd98ef851974963b6577e07bc163181 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acd98ef851974963b6577e07bc163181; no voters:
I20260928 12:22:11.767712 12544 leader_election.cc:288] T 00000000000000000000000000000000 P acd98ef851974963b6577e07bc163181 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:22:11.767735 12547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acd98ef851974963b6577e07bc163181 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:22:11.768031 12547 raft_consensus.cc:686] T 00000000000000000000000000000000 P acd98ef851974963b6577e07bc163181 [term 1 LEADER]: Becoming Leader. State: Replica: acd98ef851974963b6577e07bc163181, State: Running, Role: LEADER
I20260928 12:22:11.768096 12544 sys_catalog.cc:564] T 00000000000000000000000000000000 P acd98ef851974963b6577e07bc163181 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:22:11.768184 12547 consensus_queue.cc:237] T 00000000000000000000000000000000 P acd98ef851974963b6577e07bc163181 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acd98ef851974963b6577e07bc163181" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 42677 } }
I20260928 12:22:11.768653 12551 sys_catalog.cc:455] T 00000000000000000000000000000000 P acd98ef851974963b6577e07bc163181 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "acd98ef851974963b6577e07bc163181" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acd98ef851974963b6577e07bc163181" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 42677 } } }
I20260928 12:22:11.768661 12549 sys_catalog.cc:455] T 00000000000000000000000000000000 P acd98ef851974963b6577e07bc163181 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acd98ef851974963b6577e07bc163181. Latest consensus state: current_term: 1 leader_uuid: "acd98ef851974963b6577e07bc163181" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acd98ef851974963b6577e07bc163181" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 42677 } } }
I20260928 12:22:11.768746 12551 sys_catalog.cc:458] T 00000000000000000000000000000000 P acd98ef851974963b6577e07bc163181 [sys.catalog]: This master's current role is: LEADER
I20260928 12:22:11.768762 12549 sys_catalog.cc:458] T 00000000000000000000000000000000 P acd98ef851974963b6577e07bc163181 [sys.catalog]: This master's current role is: LEADER
W20260928 12:22:11.770279 12565 catalog_manager.cc:1558] T 00000000000000000000000000000000 P acd98ef851974963b6577e07bc163181: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:22:11.770336 12565 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:22:11.770380 12555 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:22:11.770529 12555 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:22:11.772141 12555 catalog_manager.cc:1347] Generated new cluster ID: caa90f2c9770417faa71e6f9393e5e02
I20260928 12:22:11.772189 12555 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:22:11 dist-test-slave-920p krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:22:11.778414 12555 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:22:11.779028 12555 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:22:11.779939 12550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:11.767866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60608 (local address 127.2.56.62:42677)
0928 12:22:11.768286 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:11.768335 (+ 49us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:11.768374 (+ 39us) server_negotiation.cc:407] Connection header received
0928 12:22:11.768525 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:11.768533 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:11.768597 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:11.769564 (+ 967us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:11.769664 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:11.770864 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:11.771814 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:11.772457 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:11.772590 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:11.775386 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:22:11.775392 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:22:11.775392 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:22:11.775405 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:22:11.778167 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:22:11.778617 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:22:11.778621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:22:11.778622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:22:11.778741 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:22:11.778945 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:22:11.778946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:22:11.778946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:22:11.779270 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:22:11.779447 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 12:22:11.779533 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 12:22:11.779694 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":80,"threads_started":1}
I20260928 12:22:11.783587 12555 catalog_manager.cc:5868] T 00000000000000000000000000000000 P acd98ef851974963b6577e07bc163181: Generated new TSK 0
I20260928 12:22:11.783746 12555 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab.
Entry for principal kudu/127.2.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab.
I20260928 12:22:11.810038 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.1:0
--local_ip_for_outbound_sockets=127.2.56.1
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:42677
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:34913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:22:11.898192 12570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:22:11.898432 12570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:22:11.898464 12570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:22:11.899667 12570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:22:11.899706 12570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:22:11.899741 12570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.1
I20260928 12:22:11.901170 12570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:34913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:42677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.1
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:22:11.901430 12570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:22:11.901659 12570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:22:11.906735 12576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:22:11.906774 12578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:22:11.906788 12575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:22:11.906801 12570 server_base.cc:1034] running on GCE node
I20260928 12:22:11.907109 12570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:22:11.907330 12570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:22:11.908468 12570 hybrid_clock.cc:648] HybridClock initialized: now 1790598131908465 us; error 34 us; skew 500 ppm
Sep 28 12:22:11 dist-test-slave-920p krb5kdc[12463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598131, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:22:11.911037 12570 init.cc:377] Logged in from keytab as kudu/127.2.56.1@KRBTEST.COM (short username kudu)
I20260928 12:22:11.911486 12570 webserver.cc:460] Webserver started at http://127.2.56.1:41733/ using document root <none> and password file <none>
I20260928 12:22:11.911693 12570 fs_manager.cc:359] Metadata directory not provided
I20260928 12:22:11.911741 12570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:22:11.911870 12570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:22:11.913003 12570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/data/instance:
uuid: "33f933c65a094c4b8218f556b5a64c0f"
format_stamp: "Formatted at 2026-09-28 12:22:11 on dist-test-slave-920p"
server_key: "8abfd091fe9d0e323314ace686433eb0"
server_key_iv: "50b00e80028d05b72a92eb44225e7b9b"
server_key_version: "encryptionkey@0"
I20260928 12:22:11.913393 12570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance:
uuid: "33f933c65a094c4b8218f556b5a64c0f"
format_stamp: "Formatted at 2026-09-28 12:22:11 on dist-test-slave-920p"
server_key: "8abfd091fe9d0e323314ace686433eb0"
server_key_iv: "50b00e80028d05b72a92eb44225e7b9b"
server_key_version: "encryptionkey@0"
I20260928 12:22:11.914731 12570 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:22:11.915436 12585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:11.915583 12570 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:22:11.915665 12570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/wal
uuid: "33f933c65a094c4b8218f556b5a64c0f"
format_stamp: "Formatted at 2026-09-28 12:22:11 on dist-test-slave-920p"
server_key: "8abfd091fe9d0e323314ace686433eb0"
server_key_iv: "50b00e80028d05b72a92eb44225e7b9b"
server_key_version: "encryptionkey@0"
I20260928 12:22:11.915729 12570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:22:11.927937 12570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:22:11.930094 12570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:22:11.930225 12570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:22:11.930436 12570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:22:11.930738 12570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:22:11.930771 12570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:11.930801 12570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:22:11.930824 12570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:11.936993 12570 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.1:40317
I20260928 12:22:11.937048 12698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.1:40317 every 8 connection(s)
I20260928 12:22:11.937397 12570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
Sep 28 12:22:11 dist-test-slave-920p krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598131, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:22:11.945674 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 12570
I20260928 12:22:11.945763 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance
I20260928 12:22:11.946012 2272 external_mini_cluster.cc:1428] Setting key a095fabbd4b72418193e86ccac69149a
I20260928 12:22:11.946494 12550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:11.938094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:36069 (local address 127.2.56.62:42677)
0928 12:22:11.938182 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:11.938185 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:11.938367 (+ 182us) server_negotiation.cc:407] Connection header received
0928 12:22:11.938914 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:11.938915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:11.938927 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:11.938979 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:11.939937 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:11.940492 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:11.941596 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:11.941821 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:11.941874 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:11.943941 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:22:11.943943 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:22:11.943944 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:22:11.943952 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:22:11.945059 (+ 1107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:22:11.945478 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:22:11.945480 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:22:11.945480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:22:11.945533 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:22:11.945767 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:22:11.945769 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:22:11.945769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:22:11.945940 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:22:11.946015 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 12:22:11.946316 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 12:22:11.946407 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 12:22:11.946712 12701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:11.938075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:42677 (local address 127.2.56.1:36069)
0928 12:22:11.938241 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:11.938279 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:11.938819 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:11.939051 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:11.939055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:11.939631 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:11.939841 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:11.939847 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:11.940559 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:11.940560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:11.941537 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:11.941540 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:11.941886 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:11.941887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:11.941989 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:11.942456 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:22:11.942465 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:11.943852 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:22:11.945128 (+ 1276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:22:11.945135 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:22:11.945137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:22:11.945406 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:22:11.945584 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:22:11.945585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:22:11.945585 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:22:11.945697 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:22:11.946004 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:22:11.946010 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:22:11.946164 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 12:22:11.946272 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 12:22:11.946467 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":96,"threads_started":1}
I20260928 12:22:11.947178 12699 heartbeater.cc:347] Connected to a master server at 127.2.56.62:42677
I20260928 12:22:11.947278 12699 heartbeater.cc:464] Registering TS with master...
I20260928 12:22:11.947470 12699 heartbeater.cc:511] Master 127.2.56.62:42677 requested a full tablet report, sending...
I20260928 12:22:11.948072 12508 ts_manager.cc:194] Registered new tserver with Master: 33f933c65a094c4b8218f556b5a64c0f (127.2.56.1:40317)
I20260928 12:22:11.949016 12508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.1@KRBTEST.COM'} at 127.2.56.1:36069
WARNING: no policy specified for kudu/127.2.56.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab.
Entry for principal kudu/127.2.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab.
I20260928 12:22:11.971979 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.2:0
--local_ip_for_outbound_sockets=127.2.56.2
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:42677
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:34913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:22:12.058007 12704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:22:12.058231 12704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:22:12.058264 12704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:22:12.059429 12704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:22:12.059469 12704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:22:12.059504 12704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.2
I20260928 12:22:12.060900 12704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:34913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:42677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.2
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:22:12.061138 12704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:22:12.061375 12704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:22:12.066708 12710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:22:12.066720 12712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:22:12.066845 12704 server_base.cc:1034] running on GCE node
W20260928 12:22:12.066663 12709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:22:12.067106 12704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:22:12.067404 12704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:22:12.068538 12704 hybrid_clock.cc:648] HybridClock initialized: now 1790598132068531 us; error 27 us; skew 500 ppm
Sep 28 12:22:12 dist-test-slave-920p krb5kdc[12463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598132, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:22:12.071234 12704 init.cc:377] Logged in from keytab as kudu/127.2.56.2@KRBTEST.COM (short username kudu)
I20260928 12:22:12.071668 12704 webserver.cc:460] Webserver started at http://127.2.56.2:45021/ using document root <none> and password file <none>
I20260928 12:22:12.071873 12704 fs_manager.cc:359] Metadata directory not provided
I20260928 12:22:12.071924 12704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:22:12.072052 12704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:22:12.073163 12704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/data/instance:
uuid: "a684758b347d4ebeb0ac0ad3e7d1fa85"
format_stamp: "Formatted at 2026-09-28 12:22:12 on dist-test-slave-920p"
server_key: "3bf11bfb36c22a051ed9958387d11ca5"
server_key_iv: "056a4000a2a7111403c1bc82caa77bde"
server_key_version: "encryptionkey@0"
I20260928 12:22:12.073551 12704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance:
uuid: "a684758b347d4ebeb0ac0ad3e7d1fa85"
format_stamp: "Formatted at 2026-09-28 12:22:12 on dist-test-slave-920p"
server_key: "3bf11bfb36c22a051ed9958387d11ca5"
server_key_iv: "056a4000a2a7111403c1bc82caa77bde"
server_key_version: "encryptionkey@0"
I20260928 12:22:12.074909 12704 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:22:12.075556 12719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:12.075706 12704 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:22:12.075788 12704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/wal
uuid: "a684758b347d4ebeb0ac0ad3e7d1fa85"
format_stamp: "Formatted at 2026-09-28 12:22:12 on dist-test-slave-920p"
server_key: "3bf11bfb36c22a051ed9958387d11ca5"
server_key_iv: "056a4000a2a7111403c1bc82caa77bde"
server_key_version: "encryptionkey@0"
I20260928 12:22:12.075852 12704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:22:12.095137 12704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:22:12.097540 12704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:22:12.097680 12704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:22:12.097914 12704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:22:12.098242 12704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:22:12.098275 12704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:12.098306 12704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:22:12.098330 12704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:12.104550 12704 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.2:37309
I20260928 12:22:12.104609 12832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.2:37309 every 8 connection(s)
I20260928 12:22:12.104943 12704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
I20260928 12:22:12.106986 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 12704
I20260928 12:22:12.107098 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance
I20260928 12:22:12.107316 2272 external_mini_cluster.cc:1428] Setting key 11db31d11ce8002f34f3bfa9adfb368f
Sep 28 12:22:12 dist-test-slave-920p krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598132, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:22:12.115092 12550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:12.105684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:41777 (local address 127.2.56.62:42677)
0928 12:22:12.105738 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:12.105741 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:12.105965 (+ 224us) server_negotiation.cc:407] Connection header received
0928 12:22:12.106535 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:12.106536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:12.106546 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:12.106596 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:12.107859 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:12.108445 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:12.109905 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:12.110108 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:12.110167 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:12.112432 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:22:12.112434 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:22:12.112434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:22:12.112441 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:22:12.113790 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:22:12.114151 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:22:12.114153 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:22:12.114153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:22:12.114208 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:22:12.114427 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:22:12.114428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:22:12.114428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:22:12.114542 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:22:12.114598 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 12:22:12.114895 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 12:22:12.114991 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 12:22:12.115273 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:12.105659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:42677 (local address 127.2.56.2:41777)
0928 12:22:12.105831 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:12.105877 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:12.106451 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:12.106670 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:12.106674 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:12.107470 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:12.107769 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:12.107777 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:12.108535 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:12.108536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:12.109833 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:12.109837 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:12.110176 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:12.110177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:12.110317 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:12.110871 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:22:12.110880 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:12.112343 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:22:12.113859 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:22:12.113867 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:22:12.113869 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:22:12.114087 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:22:12.114261 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:22:12.114262 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:22:12.114262 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:22:12.114374 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:22:12.114595 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:22:12.114602 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:22:12.114748 (+ 146us) client_negotiation.cc:769] Sending connection context
0928 12:22:12.114867 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 12:22:12.115033 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":73,"threads_started":1}
WARNING: no policy specified for kudu/127.2.56.3@KRBTEST.COM; defaulting to no policy
I20260928 12:22:12.115724 12833 heartbeater.cc:347] Connected to a master server at 127.2.56.62:42677
I20260928 12:22:12.115828 12833 heartbeater.cc:464] Registering TS with master...
I20260928 12:22:12.116032 12833 heartbeater.cc:511] Master 127.2.56.62:42677 requested a full tablet report, sending...
I20260928 12:22:12.116487 12508 ts_manager.cc:194] Registered new tserver with Master: a684758b347d4ebeb0ac0ad3e7d1fa85 (127.2.56.2:37309)
I20260928 12:22:12.116891 12508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.2@KRBTEST.COM'} at 127.2.56.2:41777
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab.
Entry for principal kudu/127.2.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab.
I20260928 12:22:12.131737 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.3:0
--local_ip_for_outbound_sockets=127.2.56.3
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:42677
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:34913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:22:12.217757 12838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:22:12.218029 12838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:22:12.218055 12838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:22:12.219324 12838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:22:12.219369 12838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:22:12.219396 12838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.3
I20260928 12:22:12.220911 12838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:34913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:42677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.3
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:22:12.221139 12838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:22:12.221377 12838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:22:12.226588 12843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:22:12.226639 12846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:22:12.226679 12838 server_base.cc:1034] running on GCE node
W20260928 12:22:12.226585 12844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:22:12.226975 12838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:22:12.227226 12838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:22:12.228364 12838 hybrid_clock.cc:648] HybridClock initialized: now 1790598132228347 us; error 33 us; skew 500 ppm
Sep 28 12:22:12 dist-test-slave-920p krb5kdc[12463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598132, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:22:12.230980 12838 init.cc:377] Logged in from keytab as kudu/127.2.56.3@KRBTEST.COM (short username kudu)
I20260928 12:22:12.231455 12838 webserver.cc:460] Webserver started at http://127.2.56.3:39943/ using document root <none> and password file <none>
I20260928 12:22:12.231689 12838 fs_manager.cc:359] Metadata directory not provided
I20260928 12:22:12.231738 12838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:22:12.231868 12838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:22:12.233011 12838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/data/instance:
uuid: "046fc484de924bc98a4bbfe14b380209"
format_stamp: "Formatted at 2026-09-28 12:22:12 on dist-test-slave-920p"
server_key: "48abf4717b6c36589e862bcd6470ad55"
server_key_iv: "72a993584ce4d23601d6927aa6350c50"
server_key_version: "encryptionkey@0"
I20260928 12:22:12.233425 12838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance:
uuid: "046fc484de924bc98a4bbfe14b380209"
format_stamp: "Formatted at 2026-09-28 12:22:12 on dist-test-slave-920p"
server_key: "48abf4717b6c36589e862bcd6470ad55"
server_key_iv: "72a993584ce4d23601d6927aa6350c50"
server_key_version: "encryptionkey@0"
I20260928 12:22:12.234719 12838 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:22:12.235389 12853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:12.235533 12838 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:22:12.235615 12838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/wal
uuid: "046fc484de924bc98a4bbfe14b380209"
format_stamp: "Formatted at 2026-09-28 12:22:12 on dist-test-slave-920p"
server_key: "48abf4717b6c36589e862bcd6470ad55"
server_key_iv: "72a993584ce4d23601d6927aa6350c50"
server_key_version: "encryptionkey@0"
I20260928 12:22:12.235675 12838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:22:12.252054 12838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:22:12.254441 12838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:22:12.254590 12838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:22:12.254844 12838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:22:12.255190 12838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:22:12.255224 12838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:12.255256 12838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:22:12.255280 12838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:12.261672 12838 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.3:34099
I20260928 12:22:12.261727 12966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.3:34099 every 8 connection(s)
I20260928 12:22:12.262076 12838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
I20260928 12:22:12.266820 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 12838
I20260928 12:22:12.266927 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance
I20260928 12:22:12.267112 2272 external_mini_cluster.cc:1428] Setting key 6281de5b51461c72b4ac01e74e5a877f
Sep 28 12:22:12 dist-test-slave-920p krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598132, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:22:12.271658 12550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:12.262841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:58473 (local address 127.2.56.62:42677)
0928 12:22:12.262938 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:12.262940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:12.263149 (+ 209us) server_negotiation.cc:407] Connection header received
0928 12:22:12.263788 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:12.263790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:12.263802 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:12.263847 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:12.264842 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:12.265400 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:12.266494 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:12.266695 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:12.266755 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:12.269225 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:22:12.269227 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:22:12.269227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:22:12.269234 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:22:12.270587 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:22:12.270937 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:22:12.270938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:22:12.270938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:22:12.270985 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:22:12.271163 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:22:12.271163 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:22:12.271163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:22:12.271264 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:22:12.271307 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 12:22:12.271530 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 12:22:12.271588 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 12:22:12.271839 12969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:12.262821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:42677 (local address 127.2.56.3:58473)
0928 12:22:12.263002 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:12.263056 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:12.263669 (+ 613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:12.263928 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:12.263933 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:12.264531 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:12.264769 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:12.264775 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:12.265504 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:12.265505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:12.266391 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:12.266394 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:12.266766 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:12.266768 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:12.266901 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:12.267478 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:22:12.267489 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:12.269154 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:22:12.270648 (+ 1494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:22:12.270653 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:22:12.270656 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:22:12.270867 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:22:12.271027 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:22:12.271027 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:22:12.271028 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:22:12.271122 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:22:12.271303 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:22:12.271306 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:22:12.271437 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 12:22:12.271519 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 12:22:12.271649 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
I20260928 12:22:12.272481 12967 heartbeater.cc:347] Connected to a master server at 127.2.56.62:42677
I20260928 12:22:12.272578 12967 heartbeater.cc:464] Registering TS with master...
I20260928 12:22:12.272742 12967 heartbeater.cc:511] Master 127.2.56.62:42677 requested a full tablet report, sending...
I20260928 12:22:12.273116 12508 ts_manager.cc:194] Registered new tserver with Master: 046fc484de924bc98a4bbfe14b380209 (127.2.56.3:34099)
I20260928 12:22:12.273520 12508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.3@KRBTEST.COM'} at 127.2.56.3:58473
I20260928 12:22:12.278918 2272 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:22:12.286283 12550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:12.280342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60622 (local address 127.2.56.62:42677)
0928 12:22:12.280384 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:12.280387 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:12.280523 (+ 136us) server_negotiation.cc:407] Connection header received
0928 12:22:12.280582 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:12.280583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:12.280594 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:12.280640 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:12.281173 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:12.281727 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:12.282445 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:12.282634 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:12.282687 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:12.283926 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:22:12.283928 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:22:12.283928 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:22:12.283935 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:22:12.285181 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:22:12.285510 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:22:12.285511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:22:12.285512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:22:12.285572 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:22:12.285734 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:22:12.285734 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:22:12.285735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:22:12.285903 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:22:12.286002 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 12:22:12.286106 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 12:22:12.286185 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:22:12.287832 12508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:22:12.289765 12508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:22:12.296331 12979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:12.292137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:34099 (local address 127.0.0.1:38988)
0928 12:22:12.292291 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:12.292333 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:12.292406 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:12.292865 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:12.292869 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:12.292871 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:22:12.292956 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:12.292958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:12.294011 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:12.294012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:12.295139 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:12.295141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:12.296033 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:12.296034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:12.296110 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:12.296127 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 12:22:12.296190 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 12:22:12.296249 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":50,"threads_started":1}
I20260928 12:22:12.296463 12983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:12.292516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47282 (local address 127.2.56.2:37309)
0928 12:22:12.292684 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:12.292688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:12.292751 (+ 63us) server_negotiation.cc:407] Connection header received
0928 12:22:12.292889 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:12.292892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:12.292910 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:12.293156 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:22:12.293239 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:12.294402 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:12.295563 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:12.296103 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:12.296172 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:12.296233 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 12:22:12.296326 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 12:22:12.296373 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
I20260928 12:22:12.296630 12980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:12.292335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:40317 (local address 127.0.0.1:53114)
0928 12:22:12.292921 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:12.292930 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:12.292983 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:12.293196 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:12.293197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:12.293199 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:22:12.293237 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:12.293238 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:12.294536 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:12.294537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:12.295524 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:12.295526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:12.296442 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:12.296443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:12.296494 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:12.296498 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:22:12.296537 (+ 39us) client_negotiation.cc:240] Negotiation successful
0928 12:22:12.296568 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":2,"thread_start_us":84,"threads_started":1}
I20260928 12:22:12.296756 12981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:12.292311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38988 (local address 127.2.56.3:34099)
0928 12:22:12.292710 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:12.292714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:12.292724 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:22:12.292761 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:12.292763 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:12.292777 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:12.292853 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:22:12.293122 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:12.293929 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:12.295219 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:12.295952 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:12.296025 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:12.296087 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 12:22:12.296556 (+ 469us) server_negotiation.cc:299] Negotiation successful
0928 12:22:12.296631 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":104,"threads_started":1}
I20260928 12:22:12.296790 12978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:12.292147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:37309 (local address 127.0.0.1:47282)
0928 12:22:12.292693 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:12.292702 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:12.292771 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:12.293017 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:12.293019 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:12.293021 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:22:12.293079 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:12.293081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:12.294509 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:12.294510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:12.295480 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:12.295482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:12.296174 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:12.296175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:12.296238 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:12.296243 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:22:12.296309 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 12:22:12.296353 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":24,"thread_start_us":89,"threads_started":1}
I20260928 12:22:12.297008 12982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:12.292484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53114 (local address 127.2.56.1:40317)
0928 12:22:12.292666 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:12.292672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:12.293041 (+ 369us) server_negotiation.cc:407] Connection header received
0928 12:22:12.293093 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:12.293096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:12.293112 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:12.293189 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:22:12.293337 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:12.294431 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:12.295612 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:12.296307 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:12.296360 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:12.296411 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 12:22:12.296805 (+ 394us) server_negotiation.cc:299] Negotiation successful
0928 12:22:12.296920 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":93,"threads_started":1}
I20260928 12:22:12.297609 12901 tablet_service.cc:1467] Processing CreateTablet for tablet c6ddbf19116646ed93856f2612f3afd3 (DEFAULT_TABLE table=test-table [id=f0eb2b45d3d440ae9f510efceaa11896]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:22:12.297662 12633 tablet_service.cc:1467] Processing CreateTablet for tablet c6ddbf19116646ed93856f2612f3afd3 (DEFAULT_TABLE table=test-table [id=f0eb2b45d3d440ae9f510efceaa11896]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:22:12.297963 12633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6ddbf19116646ed93856f2612f3afd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:22:12.297963 12901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6ddbf19116646ed93856f2612f3afd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:22:12.300789 12985 tablet_bootstrap.cc:492] T c6ddbf19116646ed93856f2612f3afd3 P 33f933c65a094c4b8218f556b5a64c0f: Bootstrap starting.
I20260928 12:22:12.300735 12984 tablet_bootstrap.cc:492] T c6ddbf19116646ed93856f2612f3afd3 P 046fc484de924bc98a4bbfe14b380209: Bootstrap starting.
I20260928 12:22:12.301569 12984 tablet_bootstrap.cc:654] T c6ddbf19116646ed93856f2612f3afd3 P 046fc484de924bc98a4bbfe14b380209: Neither blocks nor log segments found. Creating new log.
I20260928 12:22:12.301590 12985 tablet_bootstrap.cc:654] T c6ddbf19116646ed93856f2612f3afd3 P 33f933c65a094c4b8218f556b5a64c0f: Neither blocks nor log segments found. Creating new log.
I20260928 12:22:12.301613 12767 tablet_service.cc:1467] Processing CreateTablet for tablet c6ddbf19116646ed93856f2612f3afd3 (DEFAULT_TABLE table=test-table [id=f0eb2b45d3d440ae9f510efceaa11896]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:22:12.302009 12767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6ddbf19116646ed93856f2612f3afd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:22:12.302107 12984 log.cc:824] T c6ddbf19116646ed93856f2612f3afd3 P 046fc484de924bc98a4bbfe14b380209: Log is configured to *not* fsync() on all Append() calls
I20260928 12:22:12.302134 12985 log.cc:824] T c6ddbf19116646ed93856f2612f3afd3 P 33f933c65a094c4b8218f556b5a64c0f: Log is configured to *not* fsync() on all Append() calls
I20260928 12:22:12.302826 12985 tablet_bootstrap.cc:492] T c6ddbf19116646ed93856f2612f3afd3 P 33f933c65a094c4b8218f556b5a64c0f: No bootstrap required, opened a new log
I20260928 12:22:12.302832 12984 tablet_bootstrap.cc:492] T c6ddbf19116646ed93856f2612f3afd3 P 046fc484de924bc98a4bbfe14b380209: No bootstrap required, opened a new log
I20260928 12:22:12.302893 12985 ts_tablet_manager.cc:1397] T c6ddbf19116646ed93856f2612f3afd3 P 33f933c65a094c4b8218f556b5a64c0f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 12:22:12.302893 12984 ts_tablet_manager.cc:1397] T c6ddbf19116646ed93856f2612f3afd3 P 046fc484de924bc98a4bbfe14b380209: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:22:12.304768 12988 tablet_bootstrap.cc:492] T c6ddbf19116646ed93856f2612f3afd3 P a684758b347d4ebeb0ac0ad3e7d1fa85: Bootstrap starting.
I20260928 12:22:12.304873 12984 raft_consensus.cc:348] T c6ddbf19116646ed93856f2612f3afd3 P 046fc484de924bc98a4bbfe14b380209 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "046fc484de924bc98a4bbfe14b380209" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34099 } } peers { permanent_uuid: "a684758b347d4ebeb0ac0ad3e7d1fa85" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 37309 } } peers { permanent_uuid: "33f933c65a094c4b8218f556b5a64c0f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 40317 } }
I20260928 12:22:12.304873 12985 raft_consensus.cc:348] T c6ddbf19116646ed93856f2612f3afd3 P 33f933c65a094c4b8218f556b5a64c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "046fc484de924bc98a4bbfe14b380209" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34099 } } peers { permanent_uuid: "a684758b347d4ebeb0ac0ad3e7d1fa85" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 37309 } } peers { permanent_uuid: "33f933c65a094c4b8218f556b5a64c0f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 40317 } }
I20260928 12:22:12.305038 12985 raft_consensus.cc:374] T c6ddbf19116646ed93856f2612f3afd3 P 33f933c65a094c4b8218f556b5a64c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:22:12.305038 12984 raft_consensus.cc:374] T c6ddbf19116646ed93856f2612f3afd3 P 046fc484de924bc98a4bbfe14b380209 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:22:12.305071 12984 raft_consensus.cc:729] T c6ddbf19116646ed93856f2612f3afd3 P 046fc484de924bc98a4bbfe14b380209 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 046fc484de924bc98a4bbfe14b380209, State: Initialized, Role: FOLLOWER
I20260928 12:22:12.305071 12985 raft_consensus.cc:729] T c6ddbf19116646ed93856f2612f3afd3 P 33f933c65a094c4b8218f556b5a64c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33f933c65a094c4b8218f556b5a64c0f, State: Initialized, Role: FOLLOWER
I20260928 12:22:12.305212 12984 consensus_queue.cc:260] T c6ddbf19116646ed93856f2612f3afd3 P 046fc484de924bc98a4bbfe14b380209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "046fc484de924bc98a4bbfe14b380209" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34099 } } peers { permanent_uuid: "a684758b347d4ebeb0ac0ad3e7d1fa85" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 37309 } } peers { permanent_uuid: "33f933c65a094c4b8218f556b5a64c0f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 40317 } }
I20260928 12:22:12.305193 12985 consensus_queue.cc:260] T c6ddbf19116646ed93856f2612f3afd3 P 33f933c65a094c4b8218f556b5a64c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "046fc484de924bc98a4bbfe14b380209" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34099 } } peers { permanent_uuid: "a684758b347d4ebeb0ac0ad3e7d1fa85" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 37309 } } peers { permanent_uuid: "33f933c65a094c4b8218f556b5a64c0f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 40317 } }
I20260928 12:22:12.305374 12988 tablet_bootstrap.cc:654] T c6ddbf19116646ed93856f2612f3afd3 P a684758b347d4ebeb0ac0ad3e7d1fa85: Neither blocks nor log segments found. Creating new log.
I20260928 12:22:12.305503 12985 ts_tablet_manager.cc:1428] T c6ddbf19116646ed93856f2612f3afd3 P 33f933c65a094c4b8218f556b5a64c0f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 12:22:12.305503 12984 ts_tablet_manager.cc:1428] T c6ddbf19116646ed93856f2612f3afd3 P 046fc484de924bc98a4bbfe14b380209: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:22:12.305648 12967 heartbeater.cc:503] Master 127.2.56.62:42677 was elected leader, sending a full tablet report...
I20260928 12:22:12.305928 12988 log.cc:824] T c6ddbf19116646ed93856f2612f3afd3 P a684758b347d4ebeb0ac0ad3e7d1fa85: Log is configured to *not* fsync() on all Append() calls
I20260928 12:22:12.305954 12699 heartbeater.cc:503] Master 127.2.56.62:42677 was elected leader, sending a full tablet report...
I20260928 12:22:12.306742 12988 tablet_bootstrap.cc:492] T c6ddbf19116646ed93856f2612f3afd3 P a684758b347d4ebeb0ac0ad3e7d1fa85: No bootstrap required, opened a new log
I20260928 12:22:12.306808 12988 ts_tablet_manager.cc:1397] T c6ddbf19116646ed93856f2612f3afd3 P a684758b347d4ebeb0ac0ad3e7d1fa85: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 12:22:12.308341 12988 raft_consensus.cc:348] T c6ddbf19116646ed93856f2612f3afd3 P a684758b347d4ebeb0ac0ad3e7d1fa85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "046fc484de924bc98a4bbfe14b380209" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34099 } } peers { permanent_uuid: "a684758b347d4ebeb0ac0ad3e7d1fa85" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 37309 } } peers { permanent_uuid: "33f933c65a094c4b8218f556b5a64c0f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 40317 } }
I20260928 12:22:12.308480 12988 raft_consensus.cc:374] T c6ddbf19116646ed93856f2612f3afd3 P a684758b347d4ebeb0ac0ad3e7d1fa85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:22:12.308501 12988 raft_consensus.cc:729] T c6ddbf19116646ed93856f2612f3afd3 P a684758b347d4ebeb0ac0ad3e7d1fa85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a684758b347d4ebeb0ac0ad3e7d1fa85, State: Initialized, Role: FOLLOWER
I20260928 12:22:12.308588 12988 consensus_queue.cc:260] T c6ddbf19116646ed93856f2612f3afd3 P a684758b347d4ebeb0ac0ad3e7d1fa85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "046fc484de924bc98a4bbfe14b380209" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34099 } } peers { permanent_uuid: "a684758b347d4ebeb0ac0ad3e7d1fa85" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 37309 } } peers { permanent_uuid: "33f933c65a094c4b8218f556b5a64c0f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 40317 } }
I20260928 12:22:12.308813 12988 ts_tablet_manager.cc:1428] T c6ddbf19116646ed93856f2612f3afd3 P a684758b347d4ebeb0ac0ad3e7d1fa85: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 12:22:12.308863 12833 heartbeater.cc:503] Master 127.2.56.62:42677 was elected leader, sending a full tablet report...
W20260928 12:22:12.355552 12834 tablet.cc:2369] T c6ddbf19116646ed93856f2612f3afd3 P a684758b347d4ebeb0ac0ad3e7d1fa85: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:22:12.367795 12989 raft_consensus.cc:482] T c6ddbf19116646ed93856f2612f3afd3 P 046fc484de924bc98a4bbfe14b380209 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:22:12.367969 12989 raft_consensus.cc:504] T c6ddbf19116646ed93856f2612f3afd3 P 046fc484de924bc98a4bbfe14b380209 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "046fc484de924bc98a4bbfe14b380209" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34099 } } peers { permanent_uuid: "a684758b347d4ebeb0ac0ad3e7d1fa85" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 37309 } } peers { permanent_uuid: "33f933c65a094c4b8218f556b5a64c0f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 40317 } }
I20260928 12:22:12.368438 12989 leader_election.cc:288] T c6ddbf19116646ed93856f2612f3afd3 P 046fc484de924bc98a4bbfe14b380209 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a684758b347d4ebeb0ac0ad3e7d1fa85 (127.2.56.2:37309), 33f933c65a094c4b8218f556b5a64c0f (127.2.56.1:40317)
I20260928 12:22:12.371505 12983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:12.368693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:55351 (local address 127.2.56.2:37309)
0928 12:22:12.368748 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:12.368751 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:12.368762 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:22:12.368837 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:12.368838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:12.368853 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:12.368906 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:22:12.369086 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:12.369703 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:12.370794 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:12.371182 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:12.371251 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:12.371263 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 12:22:12.371393 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 12:22:12.371437 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 12:22:12.371503 12982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:12.368830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:34469 (local address 127.2.56.1:40317)
0928 12:22:12.368886 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:12.368888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:12.369014 (+ 126us) server_negotiation.cc:407] Connection header received
0928 12:22:12.369048 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:12.369049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:12.369063 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:12.369115 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:22:12.369255 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:12.369777 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:12.370791 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:12.371176 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:12.371237 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:12.371249 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 12:22:12.371386 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 12:22:12.371433 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 12:22:12.371503 12969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:12.368605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:37309 (local address 127.2.56.3:55351)
0928 12:22:12.368658 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:12.368672 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:12.368757 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:12.368930 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:12.368931 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:12.368933 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:22:12.369016 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:12.369018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:12.369774 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:12.369775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:12.370704 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:12.370707 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:12.371252 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:12.371254 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:12.371335 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:12.371338 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:22:12.371381 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 12:22:12.371420 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":57}
I20260928 12:22:12.371515 12993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:12.368694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:40317 (local address 127.2.56.3:34469)
0928 12:22:12.368948 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:12.368957 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:12.369007 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:12.369126 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:12.369127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:12.369128 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:22:12.369196 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:12.369198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:12.369855 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:12.369856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:12.370704 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:12.370706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:12.371246 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:12.371247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:12.371316 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:12.371319 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:22:12.371366 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 12:22:12.371410 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":3,"thread_start_us":148,"threads_started":1}
I20260928 12:22:12.371951 12653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ddbf19116646ed93856f2612f3afd3" candidate_uuid: "046fc484de924bc98a4bbfe14b380209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33f933c65a094c4b8218f556b5a64c0f" is_pre_election: true
I20260928 12:22:12.371966 12787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ddbf19116646ed93856f2612f3afd3" candidate_uuid: "046fc484de924bc98a4bbfe14b380209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a684758b347d4ebeb0ac0ad3e7d1fa85" is_pre_election: true
I20260928 12:22:12.372125 12787 raft_consensus.cc:2454] T c6ddbf19116646ed93856f2612f3afd3 P a684758b347d4ebeb0ac0ad3e7d1fa85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 046fc484de924bc98a4bbfe14b380209 in term 0.
I20260928 12:22:12.372126 12653 raft_consensus.cc:2454] T c6ddbf19116646ed93856f2612f3afd3 P 33f933c65a094c4b8218f556b5a64c0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 046fc484de924bc98a4bbfe14b380209 in term 0.
I20260928 12:22:12.372347 12854 leader_election.cc:302] T c6ddbf19116646ed93856f2612f3afd3 P 046fc484de924bc98a4bbfe14b380209 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 046fc484de924bc98a4bbfe14b380209, 33f933c65a094c4b8218f556b5a64c0f; no voters:
I20260928 12:22:12.372478 12989 raft_consensus.cc:2781] T c6ddbf19116646ed93856f2612f3afd3 P 046fc484de924bc98a4bbfe14b380209 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:22:12.372537 12989 raft_consensus.cc:482] T c6ddbf19116646ed93856f2612f3afd3 P 046fc484de924bc98a4bbfe14b380209 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:22:12.372561 12989 raft_consensus.cc:3037] T c6ddbf19116646ed93856f2612f3afd3 P 046fc484de924bc98a4bbfe14b380209 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:22:12.373397 12989 raft_consensus.cc:504] T c6ddbf19116646ed93856f2612f3afd3 P 046fc484de924bc98a4bbfe14b380209 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "046fc484de924bc98a4bbfe14b380209" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34099 } } peers { permanent_uuid: "a684758b347d4ebeb0ac0ad3e7d1fa85" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 37309 } } peers { permanent_uuid: "33f933c65a094c4b8218f556b5a64c0f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 40317 } }
I20260928 12:22:12.373517 12989 leader_election.cc:288] T c6ddbf19116646ed93856f2612f3afd3 P 046fc484de924bc98a4bbfe14b380209 [CANDIDATE]: Term 1 election: Requested vote from peers a684758b347d4ebeb0ac0ad3e7d1fa85 (127.2.56.2:37309), 33f933c65a094c4b8218f556b5a64c0f (127.2.56.1:40317)
I20260928 12:22:12.373679 12787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ddbf19116646ed93856f2612f3afd3" candidate_uuid: "046fc484de924bc98a4bbfe14b380209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a684758b347d4ebeb0ac0ad3e7d1fa85"
I20260928 12:22:12.373703 12653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ddbf19116646ed93856f2612f3afd3" candidate_uuid: "046fc484de924bc98a4bbfe14b380209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33f933c65a094c4b8218f556b5a64c0f"
I20260928 12:22:12.373769 12787 raft_consensus.cc:3037] T c6ddbf19116646ed93856f2612f3afd3 P a684758b347d4ebeb0ac0ad3e7d1fa85 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:22:12.373780 12653 raft_consensus.cc:3037] T c6ddbf19116646ed93856f2612f3afd3 P 33f933c65a094c4b8218f556b5a64c0f [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:22:12.374533 12787 raft_consensus.cc:2454] T c6ddbf19116646ed93856f2612f3afd3 P a684758b347d4ebeb0ac0ad3e7d1fa85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 046fc484de924bc98a4bbfe14b380209 in term 1.
I20260928 12:22:12.374624 12653 raft_consensus.cc:2454] T c6ddbf19116646ed93856f2612f3afd3 P 33f933c65a094c4b8218f556b5a64c0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 046fc484de924bc98a4bbfe14b380209 in term 1.
I20260928 12:22:12.374686 12854 leader_election.cc:302] T c6ddbf19116646ed93856f2612f3afd3 P 046fc484de924bc98a4bbfe14b380209 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 046fc484de924bc98a4bbfe14b380209, a684758b347d4ebeb0ac0ad3e7d1fa85; no voters:
I20260928 12:22:12.374764 12989 raft_consensus.cc:2781] T c6ddbf19116646ed93856f2612f3afd3 P 046fc484de924bc98a4bbfe14b380209 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:22:12.374935 12989 raft_consensus.cc:686] T c6ddbf19116646ed93856f2612f3afd3 P 046fc484de924bc98a4bbfe14b380209 [term 1 LEADER]: Becoming Leader. State: Replica: 046fc484de924bc98a4bbfe14b380209, State: Running, Role: LEADER
I20260928 12:22:12.375023 12989 consensus_queue.cc:237] T c6ddbf19116646ed93856f2612f3afd3 P 046fc484de924bc98a4bbfe14b380209 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "046fc484de924bc98a4bbfe14b380209" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34099 } } peers { permanent_uuid: "a684758b347d4ebeb0ac0ad3e7d1fa85" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 37309 } } peers { permanent_uuid: "33f933c65a094c4b8218f556b5a64c0f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 40317 } }
I20260928 12:22:12.375900 12507 catalog_manager.cc:5495] T c6ddbf19116646ed93856f2612f3afd3 P 046fc484de924bc98a4bbfe14b380209 reported cstate change: term changed from 0 to 1, leader changed from <none> to 046fc484de924bc98a4bbfe14b380209 (127.2.56.3). New cstate: current_term: 1 leader_uuid: "046fc484de924bc98a4bbfe14b380209" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "046fc484de924bc98a4bbfe14b380209" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a684758b347d4ebeb0ac0ad3e7d1fa85" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 37309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33f933c65a094c4b8218f556b5a64c0f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 40317 } health_report { overall_health: UNKNOWN } } }
I20260928 12:22:12.387163 12981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:12.384783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39000 (local address 127.2.56.3:34099)
0928 12:22:12.384841 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:12.384843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:12.384851 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:22:12.384888 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:12.384889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:12.384909 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:12.384971 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:22:12.385089 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:12.385625 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:12.386412 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:12.386618 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:12.386671 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:12.386810 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:22:12.386932 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:22:12.387007 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 12:22:12.387030 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 12:22:12.387053 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 12:22:12.389117 12787 raft_consensus.cc:1261] T c6ddbf19116646ed93856f2612f3afd3 P a684758b347d4ebeb0ac0ad3e7d1fa85 [term 1 FOLLOWER]: Refusing update from remote peer 046fc484de924bc98a4bbfe14b380209: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:22:12.389132 12653 raft_consensus.cc:1261] T c6ddbf19116646ed93856f2612f3afd3 P 33f933c65a094c4b8218f556b5a64c0f [term 1 FOLLOWER]: Refusing update from remote peer 046fc484de924bc98a4bbfe14b380209: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:22:12.389490 12994 consensus_queue.cc:1035] T c6ddbf19116646ed93856f2612f3afd3 P 046fc484de924bc98a4bbfe14b380209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33f933c65a094c4b8218f556b5a64c0f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:22:12.389611 12989 consensus_queue.cc:1035] T c6ddbf19116646ed93856f2612f3afd3 P 046fc484de924bc98a4bbfe14b380209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a684758b347d4ebeb0ac0ad3e7d1fa85" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 37309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:22:12.391837 12997 mvcc.cc:205] Tried to move back new op lower bound from 7334289950263083008 to 7334289950208823296. Current Snapshot: MvccSnapshot[applied={T|T < 7334289950263083008}]
I20260928 12:22:12.392483 12998 mvcc.cc:205] Tried to move back new op lower bound from 7334289950263083008 to 7334289950208823296. Current Snapshot: MvccSnapshot[applied={T|T < 7334289950263083008}]
I20260928 12:22:12.393045 12999 mvcc.cc:205] Tried to move back new op lower bound from 7334289950263083008 to 7334289950208823296. Current Snapshot: MvccSnapshot[applied={T|T < 7334289950263083008 or (T in {7334289950263083008})}]
I20260928 12:22:12.394484 12507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:22:12.394598 12507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:22:12.395362 12507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P acd98ef851974963b6577e07bc163181: Sending DeleteTablet for 3 replicas of tablet c6ddbf19116646ed93856f2612f3afd3
I20260928 12:22:12.395666 12901 tablet_service.cc:1514] Processing DeleteTablet for tablet c6ddbf19116646ed93856f2612f3afd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:22:12 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:38988
I20260928 12:22:12.395782 12633 tablet_service.cc:1514] Processing DeleteTablet for tablet c6ddbf19116646ed93856f2612f3afd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:22:12 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:53114
I20260928 12:22:12.396010 13007 tablet_replica.cc:330] T c6ddbf19116646ed93856f2612f3afd3 P 33f933c65a094c4b8218f556b5a64c0f: stopping tablet replica
I20260928 12:22:12.396026 12767 tablet_service.cc:1514] Processing DeleteTablet for tablet c6ddbf19116646ed93856f2612f3afd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:22:12 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:47282
I20260928 12:22:12.396123 13007 raft_consensus.cc:2229] T c6ddbf19116646ed93856f2612f3afd3 P 33f933c65a094c4b8218f556b5a64c0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:22:12.396238 13007 raft_consensus.cc:2258] T c6ddbf19116646ed93856f2612f3afd3 P 33f933c65a094c4b8218f556b5a64c0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:22:12.396268 13008 tablet_replica.cc:330] T c6ddbf19116646ed93856f2612f3afd3 P a684758b347d4ebeb0ac0ad3e7d1fa85: stopping tablet replica
I20260928 12:22:12.396353 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 12570
I20260928 12:22:12.396406 13008 raft_consensus.cc:2229] T c6ddbf19116646ed93856f2612f3afd3 P a684758b347d4ebeb0ac0ad3e7d1fa85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:22:12.396507 13008 raft_consensus.cc:2258] T c6ddbf19116646ed93856f2612f3afd3 P a684758b347d4ebeb0ac0ad3e7d1fa85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:22:12.397147 13008 ts_tablet_manager.cc:1905] T c6ddbf19116646ed93856f2612f3afd3 P a684758b347d4ebeb0ac0ad3e7d1fa85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:22:12.398788 13006 tablet_replica.cc:330] T c6ddbf19116646ed93856f2612f3afd3 P 046fc484de924bc98a4bbfe14b380209: stopping tablet replica
I20260928 12:22:12.398931 13006 raft_consensus.cc:2229] T c6ddbf19116646ed93856f2612f3afd3 P 046fc484de924bc98a4bbfe14b380209 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:22:12.399036 13006 raft_consensus.cc:2258] T c6ddbf19116646ed93856f2612f3afd3 P 046fc484de924bc98a4bbfe14b380209 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:22:12.399276 13008 ts_tablet_manager.cc:1918] T c6ddbf19116646ed93856f2612f3afd3 P a684758b347d4ebeb0ac0ad3e7d1fa85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:22:12.399392 13008 log.cc:1196] T c6ddbf19116646ed93856f2612f3afd3 P a684758b347d4ebeb0ac0ad3e7d1fa85: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-1/wal/wals/c6ddbf19116646ed93856f2612f3afd3
I20260928 12:22:12.399483 13006 ts_tablet_manager.cc:1905] T c6ddbf19116646ed93856f2612f3afd3 P 046fc484de924bc98a4bbfe14b380209: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:22:12.399739 13008 ts_tablet_manager.cc:1939] T c6ddbf19116646ed93856f2612f3afd3 P a684758b347d4ebeb0ac0ad3e7d1fa85: Deleting consensus metadata
I20260928 12:22:12.400243 12492 catalog_manager.cc:4841] TS a684758b347d4ebeb0ac0ad3e7d1fa85 (127.2.56.2:37309): tablet c6ddbf19116646ed93856f2612f3afd3 (table test-table [id=f0eb2b45d3d440ae9f510efceaa11896]) successfully deleted
I20260928 12:22:12.401129 13006 ts_tablet_manager.cc:1918] T c6ddbf19116646ed93856f2612f3afd3 P 046fc484de924bc98a4bbfe14b380209: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:22:12.401199 13006 log.cc:1196] T c6ddbf19116646ed93856f2612f3afd3 P 046fc484de924bc98a4bbfe14b380209: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598072638673-2272-0/minicluster-data/ts-2/wal/wals/c6ddbf19116646ed93856f2612f3afd3
I20260928 12:22:12.401477 13006 ts_tablet_manager.cc:1939] T c6ddbf19116646ed93856f2612f3afd3 P 046fc484de924bc98a4bbfe14b380209: Deleting consensus metadata
I20260928 12:22:12.401875 12494 catalog_manager.cc:4841] TS 046fc484de924bc98a4bbfe14b380209 (127.2.56.3:34099): tablet c6ddbf19116646ed93856f2612f3afd3 (table test-table [id=f0eb2b45d3d440ae9f510efceaa11896]) successfully deleted
W20260928 12:22:12.403321 12492 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.56.1:40317 (error 108)
I20260928 12:22:12.403546 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 12704
I20260928 12:22:12.404418 12980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:12.404292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:40317 (local address 127.0.0.1:53120)
0928 12:22:12.404343 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:12.404378 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.1:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 12:22:12.404563 12492 catalog_manager.cc:4584] TS 33f933c65a094c4b8218f556b5a64c0f (127.2.56.1:40317): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c6ddbf19116646ed93856f2612f3afd3: Network error: Client connection negotiation failed: client connection to 127.2.56.1:40317: connect: Connection refused (error 111)
I20260928 12:22:12.408597 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 12838
I20260928 12:22:12.412874 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 12476
2026-09-28T12:22:12Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1865 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:22:12 dist-test-slave-920p krb5kdc[13012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:22:12 dist-test-slave-920p krb5kdc[13012](info): set up 2 sockets
Sep 28 12:22:12 dist-test-slave-920p krb5kdc[13012](info): commencing operation
krb5kdc: starting...
W20260928 12:22:13.370422 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:22:13 dist-test-slave-920p krb5kdc[13012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:22:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:22:13Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.56.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.56.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.56.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/rangeradmin_127.2.56.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.56.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/rangeradmin_127.2.56.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.56.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.56.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.56.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/rangerlookup_127.2.56.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.56.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/rangerlookup_127.2.56.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.56.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.56.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.56.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/HTTP_127.2.56.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.56.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/HTTP_127.2.56.20@KRBTEST.COM.keytab.
I20260928 12:22:13.529124 2272 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskkXRk4O/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/postgres -l logfile start
2026-09-28 12:22:15.993 UTC [13042] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 12:22:15.993 UTC [13042] LOG: listening on IPv4 address "127.2.56.20", port 52011
2026-09-28 12:22:15.997 UTC [13042] LOG: listening on Unix socket "/tmp/.s.PGSQL.52011"
2026-09-28 12:22:16.019 UTC [13045] LOG: database system was shut down at 2026-09-28 12:22:14 UTC
2026-09-28 12:22:16.023 UTC [13042] LOG: database system is ready to accept connections
I20260928 12:22:16.894187 2272 mini_postgres.cc:96] Postgres bound to 52011
2026-09-28 12:22:16.901 UTC [13053] FATAL: database "slave" does not exist
127.2.56.20:52011 - accepting connections
I20260928 12:22:16.902634 2272 mini_ranger.cc:161] Starting Ranger...
I20260928 12:22:16.920339 2272 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 12:22:17.387293 2272 mini_ranger.cc:87] Created ranger Postgres database
I20260928 12:22:17.387390 2272 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ranger-admin
2026-09-28 12:22:17,678 [I] DB FLAVOR :POSTGRES
2026-09-28 12:22:17,678 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 12:22:17,678 [I] Checking connection..
2026-09-28 12:22:17,678 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:52011/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 12:22:17,947 [I] Checking connection passed.
2026-09-28 12:22:17,948 [I] --------- Verifying version history table ---------
2026-09-28 12:22:17,948 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:52011/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 12:22:18,221 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 12:22:18,221 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:52011/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 12:22:18,473 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 12:22:18,473 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 12:22:18,474 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:52011/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 12:22:18.746 UTC [13117] WARNING: there is no transaction in progress
2026-09-28 12:22:18,752 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 12:22:18,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:52011/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 12:22:18,999 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 12:22:18,999 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 12:22:18,999 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:52011/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 12:22:19,266 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:52011/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 12:22:19,517 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:52011/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-920p.c.gcp-upstream.internal','N') ;"
2026-09-28 12:22:19,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:52011/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 12:22:20,034 [I] Table x_portal_user does not exist in database ranger
2026-09-28 12:22:20,034 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:52011/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 12:22:20,303 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 12:22:20,303 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:52011/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 12:22:20,548 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:52011/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 12:22:20,816 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 12:22:20,816 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:52011/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 12:22:21.801 UTC [13225] WARNING: there is no transaction in progress
2026-09-28 12:22:21.819 UTC [13225] WARNING: there is no transaction in progress
2026-09-28 12:22:21.837 UTC [13225] WARNING: there is no transaction in progress
2026-09-28 12:22:21.854 UTC [13225] WARNING: there is no transaction in progress
2026-09-28 12:22:21.870 UTC [13225] WARNING: there is no transaction in progress
2026-09-28 12:22:21.889 UTC [13225] WARNING: there is no transaction in progress
2026-09-28 12:22:21.957 UTC [13225] WARNING: there is no transaction in progress
2026-09-28 12:22:21.964 UTC [13225] WARNING: there is no transaction in progress
2026-09-28 12:22:21.979 UTC [13225] WARNING: there is no transaction in progress
2026-09-28 12:22:21.990 UTC [13225] WARNING: there is no transaction in progress
2026-09-28 12:22:22.001 UTC [13225] WARNING: there is no transaction in progress
2026-09-28 12:22:22.016 UTC [13225] WARNING: there is no transaction in progress
2026-09-28 12:22:22.027 UTC [13225] WARNING: there is no transaction in progress
2026-09-28 12:22:22.035 UTC [13225] WARNING: there is no transaction in progress
2026-09-28 12:22:22.042 UTC [13225] WARNING: there is no transaction in progress
2026-09-28 12:22:22.048 UTC [13225] WARNING: there is no transaction in progress
2026-09-28 12:22:22,684 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 12:22:22,684 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:52011/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 12:22:22,951 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 12:22:22,952 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:52011/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 12:22:23,222 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 12:22:23,223 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:52011/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 12:22:23,475 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 12:22:23,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:52011/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 12:22:23,741 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:52011/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 12:22:23,991 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:52011/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-920p.c.gcp-upstream.internal';"
2026-09-28 12:22:24,245 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 12:22:24,246 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:52011/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 12:22:24,508 [I] DB_PATCHES have already been applied
I20260928 12:22:24.514328 2272 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ranger-admin:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskkXRk4O/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 12:22:24.514410 2272 mini_ranger.cc:193] Using host: 127.2.56.20
I20260928 12:22:24.517827 2272 mini_ranger.cc:227] Ranger admin URL: http://127.2.56.20:38875
Sep 28, 2026 12:22:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 12:22:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 12:22:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 12:22:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 12:22:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 12:22:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 12:22:25 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 12:22:25.529278 2272 mini_ranger.cc:160] Time spent starting Ranger: real 8.627s user 0.000s sys 0.010s
Sep 28, 2026 12:22:47 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 12:22:47 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 12:22:47 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 12:22:47 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 12:22:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 12:22:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 12:22:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 12:22:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 12:22:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 12:22:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 12:22:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 12:22:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 12:22:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 12:22:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 12:22:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 12:22:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 12:22:47 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 12:22:48 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 12:22:48.676940 2272 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.56.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.56.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.56.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/rangerkms_127.2.56.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.56.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/rangerkms_127.2.56.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.56.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.56.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.56.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/HTTP_127.2.56.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.56.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/HTTP_127.2.56.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 12:22:48 dist-test-slave-920p krb5kdc[13012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598168, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 12:22:48.755789 2272 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 12:22:48.764389 2272 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 12:22:49.587169 2272 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 12:22:49.587227 2272 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ranger-kms
I20260928 12:22:49.587244 2272 mini_ranger_kms.cc:227] Using postgres at 127.2.56.20:52011
2026-09-28 12:22:49,898 [I] DB FLAVOR :POSTGRES
2026-09-28 12:22:49,898 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 12:22:49,898 [I] Checking connection
2026-09-28 12:22:49,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:52011/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 12:22:50,197 [I] connection success
2026-09-28 12:22:50,197 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 12:22:50,197 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 12:22:50,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:52011/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 12:22:50,455 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 12:22:50,455 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 12:22:50,456 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 12:22:50,456 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:52011/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 12:22:50,719 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 12:22:51.163775 2272 mini_ranger_kms.cc:310] Created kms service
I20260928 12:22:51.378633 2272 mini_ranger_kms.cc:326] Created kudu user
I20260928 12:22:51.423228 2272 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 12:22:51 dist-test-slave-920p krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598168, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.56.20@KRBTEST.COM
I20260928 12:22:51.860051 2272 mini_ranger_kms.cc:384] Added ranger policy
I20260928 12:22:51.860177 2272 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ranger-kms:/tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 12:22:51.860208 2272 mini_ranger_kms.cc:240] Using host: 127.2.56.20
I20260928 12:22:51.863384 2272 mini_ranger_kms.cc:276] Ranger KMS PID: 13406
I20260928 12:22:51.863514 2272 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.56.20:49627
Sep 28, 2026 12:22:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 12:22:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 12:22:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 12:22:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 12:22:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 12:22:53 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 12:22:53.897893 2272 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.142s user 0.002s sys 0.288s
I20260928 12:22:53.898003 2272 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 12:22:53.898082 2272 mini_ranger_kms.cc:401] 127.2.56.20:49627/kms/v1/keys
Sep 28 12:22:54 dist-test-slave-920p krb5kdc[13012](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790598174, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.56.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 12:22:54 dist-test-slave-920p krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598174, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.56.20@KRBTEST.COM for HTTP/127.2.56.20@KRBTEST.COM
Sep 28 12:22:54 dist-test-slave-920p krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598174, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.56.20@KRBTEST.COM for HTTP/127.2.56.20@KRBTEST.COM
Sep 28 12:22:57 dist-test-slave-920p krb5kdc[13012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
I20260928 12:22:57.183820 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:40845
--webserver_interface=127.2.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:40781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:40845
--ranger_config_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.56.20:49627/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:22:57.305650 13456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:22:57.305857 13456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:22:57.305887 13456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:22:57.307081 13456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:22:57.307118 13456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:22:57.307132 13456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:22:57.307143 13456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:22:57.307157 13456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:22:57.308868 13456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:40781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.56.20:49627/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:40845
--ranger_config_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:40845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:22:57.309095 13456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:22:57.309340 13456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:22:57.315167 13464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:22:57.315179 13462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:22:57.315393 13461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:22:57.315290 13456 server_base.cc:1034] running on GCE node
I20260928 12:22:57.315655 13456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:22:57.315974 13456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:22:57.317152 13456 hybrid_clock.cc:648] HybridClock initialized: now 1790598177317136 us; error 37 us; skew 500 ppm
Sep 28 12:22:57 dist-test-slave-920p krb5kdc[13012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598177, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:22:57.329639 13456 init.cc:377] Logged in from keytab as kudu/127.2.56.62@KRBTEST.COM (short username kudu)
I20260928 12:22:57.330173 13456 webserver.cc:460] Webserver started at http://127.2.56.62:45643/ using document root <none> and password file <none>
I20260928 12:22:57.330416 13456 fs_manager.cc:359] Metadata directory not provided
I20260928 12:22:57.330461 13456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:22:57.330590 13456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 12:22:57 dist-test-slave-920p krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598177, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for HTTP/127.2.56.20@KRBTEST.COM
I20260928 12:22:57.375329 13456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "142415bea3c348209ec5d28407ef2296"
format_stamp: "Formatted at 2026-09-28 12:22:57 on dist-test-slave-920p"
server_key: "d93f27fd71d86a83140d5f31fcffaebc"
server_key_iv: "09dcc604d72af429e2c5b20e9f1ae30d"
server_key_version: "kuduclusterkey@0"
I20260928 12:22:57.375860 13456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "142415bea3c348209ec5d28407ef2296"
format_stamp: "Formatted at 2026-09-28 12:22:57 on dist-test-slave-920p"
server_key: "d93f27fd71d86a83140d5f31fcffaebc"
server_key_iv: "09dcc604d72af429e2c5b20e9f1ae30d"
server_key_version: "kuduclusterkey@0"
I20260928 12:22:57.377838 13456 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 12:22:57.388801 13471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:57.389000 13456 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:22:57.389096 13456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "142415bea3c348209ec5d28407ef2296"
format_stamp: "Formatted at 2026-09-28 12:22:57 on dist-test-slave-920p"
server_key: "d93f27fd71d86a83140d5f31fcffaebc"
server_key_iv: "09dcc604d72af429e2c5b20e9f1ae30d"
server_key_version: "kuduclusterkey@0"
I20260928 12:22:57.389191 13456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:22:57.410710 13456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:22:57.412526 13456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:22:57.412664 13456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:22:57.417054 13456 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:40845
I20260928 12:22:57.417078 13523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:40845 every 8 connection(s)
I20260928 12:22:57.417464 13456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:22:57.418460 13524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:22:57.420558 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 13456
I20260928 12:22:57.420657 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:22:57.420826 2272 external_mini_cluster.cc:1428] Setting key f3150dd75bf240a93e27751bd6d58496
I20260928 12:22:57.421907 13524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 142415bea3c348209ec5d28407ef2296: Bootstrap starting.
I20260928 12:22:57.422727 13524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 142415bea3c348209ec5d28407ef2296: Neither blocks nor log segments found. Creating new log.
I20260928 12:22:57.423048 13524 log.cc:824] T 00000000000000000000000000000000 P 142415bea3c348209ec5d28407ef2296: Log is configured to *not* fsync() on all Append() calls
I20260928 12:22:57.423825 13524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 142415bea3c348209ec5d28407ef2296: No bootstrap required, opened a new log
I20260928 12:22:57.426242 13524 raft_consensus.cc:348] T 00000000000000000000000000000000 P 142415bea3c348209ec5d28407ef2296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142415bea3c348209ec5d28407ef2296" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 40845 } }
I20260928 12:22:57.426441 13524 raft_consensus.cc:374] T 00000000000000000000000000000000 P 142415bea3c348209ec5d28407ef2296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:22:57.426476 13524 raft_consensus.cc:729] T 00000000000000000000000000000000 P 142415bea3c348209ec5d28407ef2296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 142415bea3c348209ec5d28407ef2296, State: Initialized, Role: FOLLOWER
I20260928 12:22:57.426647 13524 consensus_queue.cc:260] T 00000000000000000000000000000000 P 142415bea3c348209ec5d28407ef2296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142415bea3c348209ec5d28407ef2296" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 40845 } }
I20260928 12:22:57.426725 13524 raft_consensus.cc:388] T 00000000000000000000000000000000 P 142415bea3c348209ec5d28407ef2296 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:22:57.426777 13524 raft_consensus.cc:482] T 00000000000000000000000000000000 P 142415bea3c348209ec5d28407ef2296 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:22:57.426831 13524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 142415bea3c348209ec5d28407ef2296 [term 0 FOLLOWER]: Advancing to term 1
Sep 28 12:22:57 dist-test-slave-920p krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:22:57.427474 13524 raft_consensus.cc:504] T 00000000000000000000000000000000 P 142415bea3c348209ec5d28407ef2296 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142415bea3c348209ec5d28407ef2296" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 40845 } }
I20260928 12:22:57.427578 13524 leader_election.cc:302] T 00000000000000000000000000000000 P 142415bea3c348209ec5d28407ef2296 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 142415bea3c348209ec5d28407ef2296; no voters:
I20260928 12:22:57.427762 13524 leader_election.cc:288] T 00000000000000000000000000000000 P 142415bea3c348209ec5d28407ef2296 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:22:57.427810 13529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 142415bea3c348209ec5d28407ef2296 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:22:57.428017 13529 raft_consensus.cc:686] T 00000000000000000000000000000000 P 142415bea3c348209ec5d28407ef2296 [term 1 LEADER]: Becoming Leader. State: Replica: 142415bea3c348209ec5d28407ef2296, State: Running, Role: LEADER
I20260928 12:22:57.428077 13524 sys_catalog.cc:564] T 00000000000000000000000000000000 P 142415bea3c348209ec5d28407ef2296 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:22:57.428207 13529 consensus_queue.cc:237] T 00000000000000000000000000000000 P 142415bea3c348209ec5d28407ef2296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142415bea3c348209ec5d28407ef2296" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 40845 } }
I20260928 12:22:57.428656 13529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 142415bea3c348209ec5d28407ef2296 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "142415bea3c348209ec5d28407ef2296" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142415bea3c348209ec5d28407ef2296" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 40845 } } }
I20260928 12:22:57.428797 13529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 142415bea3c348209ec5d28407ef2296 [sys.catalog]: This master's current role is: LEADER
I20260928 12:22:57.428959 13529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 142415bea3c348209ec5d28407ef2296 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 142415bea3c348209ec5d28407ef2296. Latest consensus state: current_term: 1 leader_uuid: "142415bea3c348209ec5d28407ef2296" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142415bea3c348209ec5d28407ef2296" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 40845 } } }
I20260928 12:22:57.429077 13529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 142415bea3c348209ec5d28407ef2296 [sys.catalog]: This master's current role is: LEADER
I20260928 12:22:57.429337 13538 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:22:57.429415 13524 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 12:22:57.429886 13538 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:22:57.433678 13527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:57.421336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57722 (local address 127.2.56.62:40845)
0928 12:22:57.421530 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:57.421539 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:57.421564 (+ 25us) server_negotiation.cc:407] Connection header received
0928 12:22:57.421957 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:57.421965 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:57.422028 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:57.422162 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:57.422989 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:57.424138 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:57.424874 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:57.425325 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:57.425417 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:57.427610 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:22:57.427615 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:22:57.427616 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:22:57.427627 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:22:57.429503 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:22:57.429891 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:22:57.429893 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:22:57.429894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:22:57.429980 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:22:57.431264 (+ 1284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:22:57.431291 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:22:57.431292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:22:57.432096 (+ 804us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:22:57.432808 (+ 712us) server_negotiation.cc:1035] Waiting for connection context
0928 12:22:57.432908 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 12:22:57.433344 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":70,"threads_started":1}
I20260928 12:22:57.440655 13538 catalog_manager.cc:1347] Generated new cluster ID: e2aaa82095bc420697cdc452a1529235
I20260928 12:22:57.440704 13538 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:22:57.463197 13538 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:22:57.463984 13538 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:22:57.468211 13538 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 142415bea3c348209ec5d28407ef2296: Generated new TSK 0
I20260928 12:22:57.468451 13538 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab.
Entry for principal kudu/127.2.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab.
I20260928 12:22:58.313082 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.56.20:49627/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.1:0
--local_ip_for_outbound_sockets=127.2.56.1
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:40845
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:40781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:22:58.399811 13569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:22:58.400002 13569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:22:58.400032 13569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:22:58.401170 13569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:22:58.401211 13569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:22:58.401245 13569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.1
I20260928 12:22:58.402701 13569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:40781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.56.20:49627/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:40845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.1
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:22:58.402927 13569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:22:58.403146 13569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:22:58.407940 13574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:22:58.407943 13577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:22:58.408099 13575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:22:58.408545 13569 server_base.cc:1034] running on GCE node
I20260928 12:22:58.408730 13569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:22:58.408941 13569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:22:58.416285 13569 hybrid_clock.cc:648] HybridClock initialized: now 1790598178416262 us; error 35 us; skew 500 ppm
Sep 28 12:22:58 dist-test-slave-920p krb5kdc[13012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598178, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:22:58.419343 13569 init.cc:377] Logged in from keytab as kudu/127.2.56.1@KRBTEST.COM (short username kudu)
I20260928 12:22:58.419814 13569 webserver.cc:460] Webserver started at http://127.2.56.1:34107/ using document root <none> and password file <none>
I20260928 12:22:58.420023 13569 fs_manager.cc:359] Metadata directory not provided
I20260928 12:22:58.420069 13569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:22:58.420186 13569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 12:22:58 dist-test-slave-920p krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598178, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for HTTP/127.2.56.20@KRBTEST.COM
I20260928 12:22:58.437780 13569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-0/data/instance:
uuid: "b6c1f53ce0e346858c1fe05510cc7937"
format_stamp: "Formatted at 2026-09-28 12:22:58 on dist-test-slave-920p"
server_key: "549288d579eb34cd90e87ebd42df1518"
server_key_iv: "83dd9c986dbb257a6a1643b1d7ab4281"
server_key_version: "kuduclusterkey@0"
I20260928 12:22:58.438295 13569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance:
uuid: "b6c1f53ce0e346858c1fe05510cc7937"
format_stamp: "Formatted at 2026-09-28 12:22:58 on dist-test-slave-920p"
server_key: "549288d579eb34cd90e87ebd42df1518"
server_key_iv: "83dd9c986dbb257a6a1643b1d7ab4281"
server_key_version: "kuduclusterkey@0"
I20260928 12:22:58.440017 13569 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 12:22:58.452709 13584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:58.453003 13569 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:22:58.453114 13569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-0/wal
uuid: "b6c1f53ce0e346858c1fe05510cc7937"
format_stamp: "Formatted at 2026-09-28 12:22:58 on dist-test-slave-920p"
server_key: "549288d579eb34cd90e87ebd42df1518"
server_key_iv: "83dd9c986dbb257a6a1643b1d7ab4281"
server_key_version: "kuduclusterkey@0"
I20260928 12:22:58.453217 13569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:22:58.465695 13569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:22:58.468014 13569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:22:58.468169 13569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:22:58.468578 13569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:22:58.468954 13569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:22:58.469007 13569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:58.469038 13569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:22:58.469060 13569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:58.475978 13569 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.1:44219
I20260928 12:22:58.476379 13569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
I20260928 12:22:58.478539 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 13569
I20260928 12:22:58.478648 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance
I20260928 12:22:58.478856 2272 external_mini_cluster.cc:1428] Setting key 7eb8a2ff53c11ee7bac2549768f53f32
I20260928 12:22:58.481695 13699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.1:44219 every 8 connection(s)
Sep 28 12:22:58 dist-test-slave-920p krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598178, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.56.2@KRBTEST.COM; defaulting to no policy
I20260928 12:22:58.492846 13702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:58.477323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:40845 (local address 127.2.56.1:44123)
0928 12:22:58.477582 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:58.477630 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:58.479398 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:58.481880 (+ 2482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:58.481887 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:58.482391 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:58.482854 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:58.482863 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:58.483866 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:58.483867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:58.485438 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:58.485442 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:58.485999 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:58.486001 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:58.486228 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:58.487017 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:22:58.487028 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:58.489028 (+ 2000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:22:58.491096 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:22:58.491103 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:22:58.491106 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:22:58.491381 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:22:58.491584 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:22:58.491585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:22:58.491585 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:22:58.491717 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:22:58.492036 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:22:58.492042 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:22:58.492282 (+ 240us) client_negotiation.cc:769] Sending connection context
0928 12:22:58.492393 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 12:22:58.492546 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":84,"threads_started":1}
I20260928 12:22:58.493279 13703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:58.477632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:44123 (local address 127.2.56.62:40845)
0928 12:22:58.477731 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:58.477733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:58.477743 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:22:58.481644 (+ 3901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:58.481647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:58.481663 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:58.481735 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:58.482978 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:58.483757 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:58.485542 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:58.485841 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:58.485911 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:58.489143 (+ 3232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:22:58.489145 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:22:58.489145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:22:58.489153 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:22:58.491000 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:22:58.491458 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:22:58.491459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:22:58.491459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:22:58.491519 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:22:58.491781 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:22:58.491782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:22:58.491782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:22:58.491912 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:22:58.491966 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 12:22:58.493012 (+ 1046us) server_negotiation.cc:299] Negotiation successful
0928 12:22:58.493125 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":52,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.2@KRBTEST.COM" created.
I20260928 12:22:58.493888 13700 heartbeater.cc:347] Connected to a master server at 127.2.56.62:40845
I20260928 12:22:58.493995 13700 heartbeater.cc:464] Registering TS with master...
I20260928 12:22:58.494186 13700 heartbeater.cc:511] Master 127.2.56.62:40845 requested a full tablet report, sending...
I20260928 12:22:58.494674 13488 ts_manager.cc:194] Registered new tserver with Master: b6c1f53ce0e346858c1fe05510cc7937 (127.2.56.1:44219)
I20260928 12:22:58.495498 13488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.1@KRBTEST.COM'} at 127.2.56.1:44123
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab.
Entry for principal kudu/127.2.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab.
I20260928 12:22:58.509756 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.56.20:49627/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.2:0
--local_ip_for_outbound_sockets=127.2.56.2
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:40845
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:40781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:22:58.627952 13706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:22:58.628181 13706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:22:58.628216 13706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:22:58.629676 13706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:22:58.629719 13706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:22:58.629756 13706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.2
I20260928 12:22:58.631482 13706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:40781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.56.20:49627/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:40845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.2
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:22:58.631722 13706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:22:58.631979 13706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:22:58.637982 13713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:22:58.638036 13716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:22:58.638468 13714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:22:58.638923 13706 server_base.cc:1034] running on GCE node
I20260928 12:22:58.639128 13706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:22:58.639357 13706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:22:58.640522 13706 hybrid_clock.cc:648] HybridClock initialized: now 1790598178640508 us; error 26 us; skew 500 ppm
Sep 28 12:22:58 dist-test-slave-920p krb5kdc[13012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598178, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:22:58.643101 13706 init.cc:377] Logged in from keytab as kudu/127.2.56.2@KRBTEST.COM (short username kudu)
I20260928 12:22:58.643493 13706 webserver.cc:460] Webserver started at http://127.2.56.2:33547/ using document root <none> and password file <none>
I20260928 12:22:58.643690 13706 fs_manager.cc:359] Metadata directory not provided
I20260928 12:22:58.643738 13706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:22:58.643858 13706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 12:22:58 dist-test-slave-920p krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598178, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for HTTP/127.2.56.20@KRBTEST.COM
I20260928 12:22:58.656419 13706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-1/data/instance:
uuid: "546ce600c3424c1a84dbb7e434d80c7a"
format_stamp: "Formatted at 2026-09-28 12:22:58 on dist-test-slave-920p"
server_key: "bae4c7c5a57e950e717478a71edeb1c0"
server_key_iv: "69addc9f5a2fe6dfcf7533ddf4c143b7"
server_key_version: "kuduclusterkey@0"
I20260928 12:22:58.656886 13706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance:
uuid: "546ce600c3424c1a84dbb7e434d80c7a"
format_stamp: "Formatted at 2026-09-28 12:22:58 on dist-test-slave-920p"
server_key: "bae4c7c5a57e950e717478a71edeb1c0"
server_key_iv: "69addc9f5a2fe6dfcf7533ddf4c143b7"
server_key_version: "kuduclusterkey@0"
I20260928 12:22:58.664676 13706 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.002s sys 0.000s
I20260928 12:22:58.685410 13723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:58.685635 13706 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.000s
I20260928 12:22:58.685747 13706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-1/wal
uuid: "546ce600c3424c1a84dbb7e434d80c7a"
format_stamp: "Formatted at 2026-09-28 12:22:58 on dist-test-slave-920p"
server_key: "bae4c7c5a57e950e717478a71edeb1c0"
server_key_iv: "69addc9f5a2fe6dfcf7533ddf4c143b7"
server_key_version: "kuduclusterkey@0"
I20260928 12:22:58.685873 13706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 28 12:22:58 dist-test-slave-920p krb5kdc[13012](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790598178, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:22:58.719415 13706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:22:58.721629 13706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:22:58.721781 13706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:22:58.722010 13706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:22:58.722364 13706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:22:58.722414 13706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:58.722450 13706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:22:58.722469 13706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:58.728621 13706 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.2:33325
I20260928 12:22:58.729024 13706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
I20260928 12:22:58.730381 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 13706
I20260928 12:22:58.730482 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance
I20260928 12:22:58.730669 2272 external_mini_cluster.cc:1428] Setting key 90ceedef8f54bf245b5e528d34f49bea
I20260928 12:22:58.730914 13836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.2:33325 every 8 connection(s)
Sep 28 12:22:58 dist-test-slave-920p krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598178, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:22:58.744357 13703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:58.732222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:37597 (local address 127.2.56.62:40845)
0928 12:22:58.732277 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:58.732279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:58.733000 (+ 721us) server_negotiation.cc:407] Connection header received
0928 12:22:58.733700 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:58.733701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:58.733713 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:58.733743 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:58.734531 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:58.735256 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:58.736709 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:58.737010 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:58.737047 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:58.741373 (+ 4326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:22:58.741375 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:22:58.741375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:22:58.741383 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:22:58.742950 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:22:58.743380 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:22:58.743381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:22:58.743381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:22:58.743431 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:22:58.743675 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:22:58.743676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:22:58.743676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:22:58.743797 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:22:58.743835 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 12:22:58.744172 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 12:22:58.744255 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 12:22:58.744849 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:58.731950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:40845 (local address 127.2.56.2:37597)
0928 12:22:58.732889 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:58.732928 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:58.733585 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:58.733918 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:58.733922 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:58.734148 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:58.734410 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:58.734416 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:58.735372 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:58.735373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:58.736617 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:58.736619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:58.737127 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:58.737127 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:58.737259 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:58.739766 (+ 2507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:22:58.739776 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:58.741232 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:22:58.743033 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:22:58.743040 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:22:58.743042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:22:58.743302 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:22:58.743501 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:22:58.743501 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:22:58.743501 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:22:58.743612 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:22:58.743872 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:22:58.743876 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:22:58.744036 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 12:22:58.744482 (+ 446us) client_negotiation.cc:240] Negotiation successful
0928 12:22:58.744638 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"thread_start_us":88,"threads_started":1}
I20260928 12:22:58.745258 13837 heartbeater.cc:347] Connected to a master server at 127.2.56.62:40845
I20260928 12:22:58.745385 13837 heartbeater.cc:464] Registering TS with master...
I20260928 12:22:58.745573 13837 heartbeater.cc:511] Master 127.2.56.62:40845 requested a full tablet report, sending...
I20260928 12:22:58.745893 13488 ts_manager.cc:194] Registered new tserver with Master: 546ce600c3424c1a84dbb7e434d80c7a (127.2.56.2:33325)
I20260928 12:22:58.746387 13488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.2@KRBTEST.COM'} at 127.2.56.2:37597
WARNING: no policy specified for kudu/127.2.56.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab.
Entry for principal kudu/127.2.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab.
I20260928 12:22:58.770179 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.56.20:49627/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.3:0
--local_ip_for_outbound_sockets=127.2.56.3
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:40845
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:40781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:22:58.896723 13843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:22:58.897022 13843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:22:58.897061 13843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:22:58.898881 13843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:22:58.898934 13843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:22:58.898974 13843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.3
I20260928 12:22:58.901211 13843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:40781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.56.20:49627/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:40845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.3
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:22:58.901497 13843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:22:58.901757 13843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:22:58.907959 13851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:22:58.907966 13849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:22:58.908357 13848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:22:58.908516 13843 server_base.cc:1034] running on GCE node
I20260928 12:22:58.908820 13843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:22:58.909093 13843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:22:58.910251 13843 hybrid_clock.cc:648] HybridClock initialized: now 1790598178910228 us; error 36 us; skew 500 ppm
Sep 28 12:22:58 dist-test-slave-920p krb5kdc[13012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598178, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:22:58.913468 13843 init.cc:377] Logged in from keytab as kudu/127.2.56.3@KRBTEST.COM (short username kudu)
I20260928 12:22:58.913951 13843 webserver.cc:460] Webserver started at http://127.2.56.3:37687/ using document root <none> and password file <none>
I20260928 12:22:58.914208 13843 fs_manager.cc:359] Metadata directory not provided
I20260928 12:22:58.914268 13843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:22:58.914390 13843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 12:22:58 dist-test-slave-920p krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598178, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for HTTP/127.2.56.20@KRBTEST.COM
I20260928 12:22:58.933061 13843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-2/data/instance:
uuid: "ac283b2d670b4aefb9441cf97a9fe03b"
format_stamp: "Formatted at 2026-09-28 12:22:58 on dist-test-slave-920p"
server_key: "ee3379d63fa8ba48ca8a1410278d47fe"
server_key_iv: "a6aaa85130b5ed63ae154a7054a92df0"
server_key_version: "kuduclusterkey@0"
I20260928 12:22:58.933645 13843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance:
uuid: "ac283b2d670b4aefb9441cf97a9fe03b"
format_stamp: "Formatted at 2026-09-28 12:22:58 on dist-test-slave-920p"
server_key: "ee3379d63fa8ba48ca8a1410278d47fe"
server_key_iv: "a6aaa85130b5ed63ae154a7054a92df0"
server_key_version: "kuduclusterkey@0"
I20260928 12:22:58.941745 13843 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.001s sys 0.000s
I20260928 12:22:58.958684 13858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:58.965255 13843 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.000s
I20260928 12:22:58.965425 13843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-2/wal
uuid: "ac283b2d670b4aefb9441cf97a9fe03b"
format_stamp: "Formatted at 2026-09-28 12:22:58 on dist-test-slave-920p"
server_key: "ee3379d63fa8ba48ca8a1410278d47fe"
server_key_iv: "a6aaa85130b5ed63ae154a7054a92df0"
server_key_version: "kuduclusterkey@0"
I20260928 12:22:58.965543 13843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:22:58.976193 13843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:22:58.978475 13843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:22:58.978709 13843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:22:58.979076 13843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:22:58.979522 13843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:22:58.979578 13843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:58.979611 13843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:22:58.979628 13843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:22:58.989593 13843 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.3:40817
I20260928 12:22:58.989773 13971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.3:40817 every 8 connection(s)
I20260928 12:22:58.990049 13843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
I20260928 12:22:58.997345 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 13843
I20260928 12:22:58.997433 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance
I20260928 12:22:58.997603 2272 external_mini_cluster.cc:1428] Setting key c41953fc15829062e0a03e3a0da76dd4
Sep 28 12:22:58 dist-test-slave-920p krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598178, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:22:59.002139 13703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:58.990968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:41745 (local address 127.2.56.62:40845)
0928 12:22:58.991032 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:58.991035 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:58.991384 (+ 349us) server_negotiation.cc:407] Connection header received
0928 12:22:58.992025 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:58.992026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:58.992076 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:58.992179 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:58.992902 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:58.994198 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:58.995859 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:58.996156 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:58.996201 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:58.999364 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:22:58.999365 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:22:58.999366 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:22:58.999374 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:22:59.000866 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:22:59.001231 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:22:59.001232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:22:59.001232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:22:59.001315 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:22:59.001529 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:22:59.001530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:22:59.001530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:22:59.001668 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:22:59.001755 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 12:22:59.001962 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 12:22:59.002043 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 12:22:59.002230 13974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:58.990925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:40845 (local address 127.2.56.3:41745)
0928 12:22:58.991193 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:58.991244 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:58.991917 (+ 673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:58.992258 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:58.992262 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:58.992520 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:22:58.992806 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:58.992813 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:58.994302 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:58.994304 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:58.995745 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:58.995748 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:58.996309 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:58.996310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:58.996518 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:58.997120 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:22:58.997130 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:22:58.999249 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:22:59.000942 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:22:59.000948 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:22:59.000950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:22:59.001164 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:22:59.001376 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:22:59.001377 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:22:59.001377 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:22:59.001474 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:22:59.001725 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:22:59.001728 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:22:59.001852 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 12:22:59.001929 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 12:22:59.002043 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":179,"threads_started":1}
I20260928 12:22:59.002756 13972 heartbeater.cc:347] Connected to a master server at 127.2.56.62:40845
I20260928 12:22:59.002839 13972 heartbeater.cc:464] Registering TS with master...
I20260928 12:22:59.003005 13972 heartbeater.cc:511] Master 127.2.56.62:40845 requested a full tablet report, sending...
I20260928 12:22:59.003361 13488 ts_manager.cc:194] Registered new tserver with Master: ac283b2d670b4aefb9441cf97a9fe03b (127.2.56.3:40817)
I20260928 12:22:59.003855 13488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.3@KRBTEST.COM'} at 127.2.56.3:41745
I20260928 12:22:59.008993 2272 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:22:59.016737 13703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:59.010535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57728 (local address 127.2.56.62:40845)
0928 12:22:59.010580 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:59.010582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:59.010736 (+ 154us) server_negotiation.cc:407] Connection header received
0928 12:22:59.010783 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:59.010784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:59.010796 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:59.010862 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:22:59.011312 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:59.012033 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:59.012710 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:59.012999 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:59.013068 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:59.014140 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:22:59.014142 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:22:59.014143 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:22:59.014150 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:22:59.015697 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:22:59.015989 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:22:59.015990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:22:59.015990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:22:59.016051 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:22:59.016260 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:22:59.016261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:22:59.016262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:22:59.016389 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:22:59.016487 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 12:22:59.016565 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 12:22:59.016644 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 12:22:59.018302 13488 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:22:59.020748 13488 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:22:59.030732 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:59.023677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52164 (local address 127.2.56.2:33325)
0928 12:22:59.024179 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:59.024182 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:59.025930 (+ 1748us) server_negotiation.cc:407] Connection header received
0928 12:22:59.025977 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:59.025979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:59.025996 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:59.026063 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:22:59.026550 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:59.027654 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:59.029328 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:59.030065 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:59.030113 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:59.030225 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 12:22:59.030551 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 12:22:59.030589 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":73,"threads_started":1}
I20260928 12:22:59.030992 13988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:59.023523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:33325 (local address 127.0.0.1:52164)
0928 12:22:59.025862 (+ 2339us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:59.025871 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:59.025919 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:59.026140 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:59.026141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:59.026143 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:22:59.026469 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:59.026472 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:59.027753 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:59.027754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:59.029204 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:59.029206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:59.030306 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:59.030307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:59.030426 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:59.030442 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 12:22:59.030848 (+ 406us) client_negotiation.cc:240] Negotiation successful
0928 12:22:59.030920 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2302,"mutex_wait_us":29,"spinlock_wait_cycles":768,"thread_start_us":333,"threads_started":1}
I20260928 12:22:59.031831 13771 tablet_service.cc:1467] Processing CreateTablet for tablet 162d8e0051e2495c9b89b69d3c8bf4d2 (DEFAULT_TABLE table=test-table [id=dac0a51c63ed41c582dd973694c86b82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:22:59.032225 13771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 162d8e0051e2495c9b89b69d3c8bf4d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:22:59.039537 13986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:59.023522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:40817 (local address 127.0.0.1:39620)
0928 12:22:59.025681 (+ 2159us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:59.025690 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:59.025734 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:59.026293 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:59.026294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:59.026296 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:22:59.026329 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:59.026330 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:59.027840 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:59.027841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:59.029047 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:59.029049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:59.038968 (+ 9919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:59.038970 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:59.039052 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:59.039056 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:22:59.039087 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 12:22:59.039133 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2111,"thread_start_us":443,"threads_started":1}
I20260928 12:22:59.039801 13990 tablet_bootstrap.cc:492] T 162d8e0051e2495c9b89b69d3c8bf4d2 P 546ce600c3424c1a84dbb7e434d80c7a: Bootstrap starting.
I20260928 12:22:59.040401 13989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:59.023769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39620 (local address 127.2.56.3:40817)
0928 12:22:59.026108 (+ 2339us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:59.026112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:59.026120 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:22:59.026180 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:59.026182 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:59.026197 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:59.026420 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:22:59.026543 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:59.027764 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:59.029245 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:59.038860 (+ 9615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:59.039827 (+ 967us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:59.040116 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0928 12:22:59.040152 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 12:22:59.040223 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2280,"thread_start_us":211,"threads_started":1}
I20260928 12:22:59.041218 13987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:59.023522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:44219 (local address 127.0.0.1:56234)
0928 12:22:59.025432 (+ 1910us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:59.025476 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:59.025557 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:59.025816 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:59.025819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:59.025822 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:22:59.025897 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:59.025899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:59.027392 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:59.027393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:59.028983 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:59.028986 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:59.040965 (+ 11979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:59.040966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:59.041041 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:59.041047 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 12:22:59.041096 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 12:22:59.041142 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1848,"mutex_wait_us":30,"thread_start_us":413,"threads_started":1}
I20260928 12:22:59.041617 13984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:59.023669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56234 (local address 127.2.56.1:44219)
0928 12:22:59.024262 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:59.024266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:59.025549 (+ 1283us) server_negotiation.cc:407] Connection header received
0928 12:22:59.025709 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:59.025712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:59.025730 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:59.025803 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:22:59.025990 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:59.027296 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:59.029075 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:59.040664 (+ 11589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:59.040748 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:59.041467 (+ 719us) server_negotiation.cc:1035] Waiting for connection context
0928 12:22:59.041496 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 12:22:59.041546 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":72,"threads_started":1}
I20260928 12:22:59.041752 13990 tablet_bootstrap.cc:654] T 162d8e0051e2495c9b89b69d3c8bf4d2 P 546ce600c3424c1a84dbb7e434d80c7a: Neither blocks nor log segments found. Creating new log.
I20260928 12:22:59.042335 13634 tablet_service.cc:1467] Processing CreateTablet for tablet 162d8e0051e2495c9b89b69d3c8bf4d2 (DEFAULT_TABLE table=test-table [id=dac0a51c63ed41c582dd973694c86b82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:22:59.042693 13990 log.cc:824] T 162d8e0051e2495c9b89b69d3c8bf4d2 P 546ce600c3424c1a84dbb7e434d80c7a: Log is configured to *not* fsync() on all Append() calls
I20260928 12:22:59.042716 13634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 162d8e0051e2495c9b89b69d3c8bf4d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:22:59.043637 13990 tablet_bootstrap.cc:492] T 162d8e0051e2495c9b89b69d3c8bf4d2 P 546ce600c3424c1a84dbb7e434d80c7a: No bootstrap required, opened a new log
I20260928 12:22:59.043713 13990 ts_tablet_manager.cc:1397] T 162d8e0051e2495c9b89b69d3c8bf4d2 P 546ce600c3424c1a84dbb7e434d80c7a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260928 12:22:59.046196 13906 tablet_service.cc:1467] Processing CreateTablet for tablet 162d8e0051e2495c9b89b69d3c8bf4d2 (DEFAULT_TABLE table=test-table [id=dac0a51c63ed41c582dd973694c86b82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:22:59.046559 13906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 162d8e0051e2495c9b89b69d3c8bf4d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:22:59.048017 13990 raft_consensus.cc:348] T 162d8e0051e2495c9b89b69d3c8bf4d2 P 546ce600c3424c1a84dbb7e434d80c7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac283b2d670b4aefb9441cf97a9fe03b" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 40817 } } peers { permanent_uuid: "546ce600c3424c1a84dbb7e434d80c7a" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 33325 } } peers { permanent_uuid: "b6c1f53ce0e346858c1fe05510cc7937" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44219 } }
I20260928 12:22:59.048195 13990 raft_consensus.cc:374] T 162d8e0051e2495c9b89b69d3c8bf4d2 P 546ce600c3424c1a84dbb7e434d80c7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:22:59.048237 13990 raft_consensus.cc:729] T 162d8e0051e2495c9b89b69d3c8bf4d2 P 546ce600c3424c1a84dbb7e434d80c7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 546ce600c3424c1a84dbb7e434d80c7a, State: Initialized, Role: FOLLOWER
I20260928 12:22:59.048373 13990 consensus_queue.cc:260] T 162d8e0051e2495c9b89b69d3c8bf4d2 P 546ce600c3424c1a84dbb7e434d80c7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac283b2d670b4aefb9441cf97a9fe03b" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 40817 } } peers { permanent_uuid: "546ce600c3424c1a84dbb7e434d80c7a" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 33325 } } peers { permanent_uuid: "b6c1f53ce0e346858c1fe05510cc7937" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44219 } }
I20260928 12:22:59.048707 13990 ts_tablet_manager.cc:1428] T 162d8e0051e2495c9b89b69d3c8bf4d2 P 546ce600c3424c1a84dbb7e434d80c7a: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260928 12:22:59.048833 13837 heartbeater.cc:503] Master 127.2.56.62:40845 was elected leader, sending a full tablet report...
I20260928 12:22:59.054220 13992 tablet_bootstrap.cc:492] T 162d8e0051e2495c9b89b69d3c8bf4d2 P b6c1f53ce0e346858c1fe05510cc7937: Bootstrap starting.
I20260928 12:22:59.056105 13992 tablet_bootstrap.cc:654] T 162d8e0051e2495c9b89b69d3c8bf4d2 P b6c1f53ce0e346858c1fe05510cc7937: Neither blocks nor log segments found. Creating new log.
I20260928 12:22:59.057096 13992 log.cc:824] T 162d8e0051e2495c9b89b69d3c8bf4d2 P b6c1f53ce0e346858c1fe05510cc7937: Log is configured to *not* fsync() on all Append() calls
I20260928 12:22:59.060037 13992 tablet_bootstrap.cc:492] T 162d8e0051e2495c9b89b69d3c8bf4d2 P b6c1f53ce0e346858c1fe05510cc7937: No bootstrap required, opened a new log
I20260928 12:22:59.060118 13992 ts_tablet_manager.cc:1397] T 162d8e0051e2495c9b89b69d3c8bf4d2 P b6c1f53ce0e346858c1fe05510cc7937: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260928 12:22:59.061610 13994 tablet_bootstrap.cc:492] T 162d8e0051e2495c9b89b69d3c8bf4d2 P ac283b2d670b4aefb9441cf97a9fe03b: Bootstrap starting.
I20260928 12:22:59.062826 13992 raft_consensus.cc:348] T 162d8e0051e2495c9b89b69d3c8bf4d2 P b6c1f53ce0e346858c1fe05510cc7937 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac283b2d670b4aefb9441cf97a9fe03b" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 40817 } } peers { permanent_uuid: "546ce600c3424c1a84dbb7e434d80c7a" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 33325 } } peers { permanent_uuid: "b6c1f53ce0e346858c1fe05510cc7937" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44219 } }
I20260928 12:22:59.063014 13992 raft_consensus.cc:374] T 162d8e0051e2495c9b89b69d3c8bf4d2 P b6c1f53ce0e346858c1fe05510cc7937 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:22:59.063061 13992 raft_consensus.cc:729] T 162d8e0051e2495c9b89b69d3c8bf4d2 P b6c1f53ce0e346858c1fe05510cc7937 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6c1f53ce0e346858c1fe05510cc7937, State: Initialized, Role: FOLLOWER
I20260928 12:22:59.063206 13992 consensus_queue.cc:260] T 162d8e0051e2495c9b89b69d3c8bf4d2 P b6c1f53ce0e346858c1fe05510cc7937 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac283b2d670b4aefb9441cf97a9fe03b" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 40817 } } peers { permanent_uuid: "546ce600c3424c1a84dbb7e434d80c7a" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 33325 } } peers { permanent_uuid: "b6c1f53ce0e346858c1fe05510cc7937" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44219 } }
I20260928 12:22:59.063482 13992 ts_tablet_manager.cc:1428] T 162d8e0051e2495c9b89b69d3c8bf4d2 P b6c1f53ce0e346858c1fe05510cc7937: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 12:22:59.063576 13994 tablet_bootstrap.cc:654] T 162d8e0051e2495c9b89b69d3c8bf4d2 P ac283b2d670b4aefb9441cf97a9fe03b: Neither blocks nor log segments found. Creating new log.
I20260928 12:22:59.063611 13700 heartbeater.cc:503] Master 127.2.56.62:40845 was elected leader, sending a full tablet report...
I20260928 12:22:59.064294 13994 log.cc:824] T 162d8e0051e2495c9b89b69d3c8bf4d2 P ac283b2d670b4aefb9441cf97a9fe03b: Log is configured to *not* fsync() on all Append() calls
I20260928 12:22:59.065966 13994 tablet_bootstrap.cc:492] T 162d8e0051e2495c9b89b69d3c8bf4d2 P ac283b2d670b4aefb9441cf97a9fe03b: No bootstrap required, opened a new log
I20260928 12:22:59.066035 13994 ts_tablet_manager.cc:1397] T 162d8e0051e2495c9b89b69d3c8bf4d2 P ac283b2d670b4aefb9441cf97a9fe03b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 12:22:59.068140 13994 raft_consensus.cc:348] T 162d8e0051e2495c9b89b69d3c8bf4d2 P ac283b2d670b4aefb9441cf97a9fe03b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac283b2d670b4aefb9441cf97a9fe03b" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 40817 } } peers { permanent_uuid: "546ce600c3424c1a84dbb7e434d80c7a" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 33325 } } peers { permanent_uuid: "b6c1f53ce0e346858c1fe05510cc7937" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44219 } }
I20260928 12:22:59.068305 13994 raft_consensus.cc:374] T 162d8e0051e2495c9b89b69d3c8bf4d2 P ac283b2d670b4aefb9441cf97a9fe03b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:22:59.068352 13994 raft_consensus.cc:729] T 162d8e0051e2495c9b89b69d3c8bf4d2 P ac283b2d670b4aefb9441cf97a9fe03b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac283b2d670b4aefb9441cf97a9fe03b, State: Initialized, Role: FOLLOWER
I20260928 12:22:59.068487 13994 consensus_queue.cc:260] T 162d8e0051e2495c9b89b69d3c8bf4d2 P ac283b2d670b4aefb9441cf97a9fe03b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac283b2d670b4aefb9441cf97a9fe03b" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 40817 } } peers { permanent_uuid: "546ce600c3424c1a84dbb7e434d80c7a" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 33325 } } peers { permanent_uuid: "b6c1f53ce0e346858c1fe05510cc7937" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44219 } }
I20260928 12:22:59.068748 13994 ts_tablet_manager.cc:1428] T 162d8e0051e2495c9b89b69d3c8bf4d2 P ac283b2d670b4aefb9441cf97a9fe03b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:22:59.068789 13972 heartbeater.cc:503] Master 127.2.56.62:40845 was elected leader, sending a full tablet report...
I20260928 12:22:59.080480 13996 raft_consensus.cc:482] T 162d8e0051e2495c9b89b69d3c8bf4d2 P b6c1f53ce0e346858c1fe05510cc7937 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:22:59.080610 13996 raft_consensus.cc:504] T 162d8e0051e2495c9b89b69d3c8bf4d2 P b6c1f53ce0e346858c1fe05510cc7937 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac283b2d670b4aefb9441cf97a9fe03b" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 40817 } } peers { permanent_uuid: "546ce600c3424c1a84dbb7e434d80c7a" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 33325 } } peers { permanent_uuid: "b6c1f53ce0e346858c1fe05510cc7937" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44219 } }
I20260928 12:22:59.080997 13996 leader_election.cc:288] T 162d8e0051e2495c9b89b69d3c8bf4d2 P b6c1f53ce0e346858c1fe05510cc7937 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac283b2d670b4aefb9441cf97a9fe03b (127.2.56.3:40817), 546ce600c3424c1a84dbb7e434d80c7a (127.2.56.2:33325)
I20260928 12:22:59.085685 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:59.081840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:41731 (local address 127.2.56.2:33325)
0928 12:22:59.082034 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:59.082036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:59.082148 (+ 112us) server_negotiation.cc:407] Connection header received
0928 12:22:59.082237 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:59.082238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:59.082248 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:59.082274 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:22:59.082528 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:59.083247 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:59.084689 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:59.085205 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:59.085254 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:59.085317 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 12:22:59.085566 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 12:22:59.085610 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260928 12:22:59.085889 13999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:59.081496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:33325 (local address 127.2.56.1:41731)
0928 12:22:59.082099 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:59.082109 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:59.082169 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:59.082338 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:59.082339 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:59.082340 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:22:59.082441 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:59.082443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:59.083373 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:59.083373 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:59.084588 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:59.084592 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:59.085380 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:59.085381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:59.085478 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:59.085483 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:22:59.085764 (+ 281us) client_negotiation.cc:240] Negotiation successful
0928 12:22:59.085807 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":63,"threads_started":1}
I20260928 12:22:59.086912 13791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "162d8e0051e2495c9b89b69d3c8bf4d2" candidate_uuid: "b6c1f53ce0e346858c1fe05510cc7937" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "546ce600c3424c1a84dbb7e434d80c7a" is_pre_election: true
I20260928 12:22:59.087070 13791 raft_consensus.cc:2454] T 162d8e0051e2495c9b89b69d3c8bf4d2 P 546ce600c3424c1a84dbb7e434d80c7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6c1f53ce0e346858c1fe05510cc7937 in term 0.
I20260928 12:22:59.087287 13588 leader_election.cc:302] T 162d8e0051e2495c9b89b69d3c8bf4d2 P b6c1f53ce0e346858c1fe05510cc7937 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 546ce600c3424c1a84dbb7e434d80c7a, b6c1f53ce0e346858c1fe05510cc7937; no voters:
I20260928 12:22:59.087423 13996 raft_consensus.cc:2781] T 162d8e0051e2495c9b89b69d3c8bf4d2 P b6c1f53ce0e346858c1fe05510cc7937 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:22:59.087463 13996 raft_consensus.cc:482] T 162d8e0051e2495c9b89b69d3c8bf4d2 P b6c1f53ce0e346858c1fe05510cc7937 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:22:59.087483 13996 raft_consensus.cc:3037] T 162d8e0051e2495c9b89b69d3c8bf4d2 P b6c1f53ce0e346858c1fe05510cc7937 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:22:59.088338 13996 raft_consensus.cc:504] T 162d8e0051e2495c9b89b69d3c8bf4d2 P b6c1f53ce0e346858c1fe05510cc7937 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac283b2d670b4aefb9441cf97a9fe03b" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 40817 } } peers { permanent_uuid: "546ce600c3424c1a84dbb7e434d80c7a" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 33325 } } peers { permanent_uuid: "b6c1f53ce0e346858c1fe05510cc7937" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44219 } }
I20260928 12:22:59.088477 13996 leader_election.cc:288] T 162d8e0051e2495c9b89b69d3c8bf4d2 P b6c1f53ce0e346858c1fe05510cc7937 [CANDIDATE]: Term 1 election: Requested vote from peers ac283b2d670b4aefb9441cf97a9fe03b (127.2.56.3:40817), 546ce600c3424c1a84dbb7e434d80c7a (127.2.56.2:33325)
I20260928 12:22:59.088518 13989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:59.081789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:35511 (local address 127.2.56.3:40817)
0928 12:22:59.081885 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:59.081887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:59.081895 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:22:59.081920 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:59.081921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:59.081932 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:59.081983 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:22:59.085494 (+ 3511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:59.086196 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:59.087597 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:59.088145 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:59.088190 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:59.088199 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 12:22:59.088400 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 12:22:59.088441 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 12:22:59.088716 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:59.081246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:40817 (local address 127.2.56.1:35511)
0928 12:22:59.081643 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:59.081655 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:22:59.081709 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:22:59.085346 (+ 3637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:22:59.085347 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:22:59.085349 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:22:59.085415 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:59.085416 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:59.086306 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:59.086307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:59.087501 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:22:59.087503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:22:59.088257 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:22:59.088258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:22:59.088326 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:59.088329 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:22:59.088588 (+ 259us) client_negotiation.cc:240] Negotiation successful
0928 12:22:59.088627 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":102,"threads_started":1}
I20260928 12:22:59.089105 13791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "162d8e0051e2495c9b89b69d3c8bf4d2" candidate_uuid: "b6c1f53ce0e346858c1fe05510cc7937" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "546ce600c3424c1a84dbb7e434d80c7a"
I20260928 12:22:59.089186 13791 raft_consensus.cc:3037] T 162d8e0051e2495c9b89b69d3c8bf4d2 P 546ce600c3424c1a84dbb7e434d80c7a [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:22:59.089164 13926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "162d8e0051e2495c9b89b69d3c8bf4d2" candidate_uuid: "b6c1f53ce0e346858c1fe05510cc7937" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac283b2d670b4aefb9441cf97a9fe03b" is_pre_election: true
I20260928 12:22:59.089327 13926 raft_consensus.cc:2454] T 162d8e0051e2495c9b89b69d3c8bf4d2 P ac283b2d670b4aefb9441cf97a9fe03b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6c1f53ce0e346858c1fe05510cc7937 in term 0.
I20260928 12:22:59.089447 13925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "162d8e0051e2495c9b89b69d3c8bf4d2" candidate_uuid: "b6c1f53ce0e346858c1fe05510cc7937" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac283b2d670b4aefb9441cf97a9fe03b"
I20260928 12:22:59.089519 13925 raft_consensus.cc:3037] T 162d8e0051e2495c9b89b69d3c8bf4d2 P ac283b2d670b4aefb9441cf97a9fe03b [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:22:59.090049 13791 raft_consensus.cc:2454] T 162d8e0051e2495c9b89b69d3c8bf4d2 P 546ce600c3424c1a84dbb7e434d80c7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6c1f53ce0e346858c1fe05510cc7937 in term 1.
I20260928 12:22:59.090233 13588 leader_election.cc:302] T 162d8e0051e2495c9b89b69d3c8bf4d2 P b6c1f53ce0e346858c1fe05510cc7937 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 546ce600c3424c1a84dbb7e434d80c7a, b6c1f53ce0e346858c1fe05510cc7937; no voters:
I20260928 12:22:59.090303 13996 raft_consensus.cc:2781] T 162d8e0051e2495c9b89b69d3c8bf4d2 P b6c1f53ce0e346858c1fe05510cc7937 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:22:59.090298 13925 raft_consensus.cc:2454] T 162d8e0051e2495c9b89b69d3c8bf4d2 P ac283b2d670b4aefb9441cf97a9fe03b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6c1f53ce0e346858c1fe05510cc7937 in term 1.
I20260928 12:22:59.090449 13996 raft_consensus.cc:686] T 162d8e0051e2495c9b89b69d3c8bf4d2 P b6c1f53ce0e346858c1fe05510cc7937 [term 1 LEADER]: Becoming Leader. State: Replica: b6c1f53ce0e346858c1fe05510cc7937, State: Running, Role: LEADER
I20260928 12:22:59.090533 13996 consensus_queue.cc:237] T 162d8e0051e2495c9b89b69d3c8bf4d2 P b6c1f53ce0e346858c1fe05510cc7937 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac283b2d670b4aefb9441cf97a9fe03b" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 40817 } } peers { permanent_uuid: "546ce600c3424c1a84dbb7e434d80c7a" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 33325 } } peers { permanent_uuid: "b6c1f53ce0e346858c1fe05510cc7937" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44219 } }
I20260928 12:22:59.091450 13488 catalog_manager.cc:5495] T 162d8e0051e2495c9b89b69d3c8bf4d2 P b6c1f53ce0e346858c1fe05510cc7937 reported cstate change: term changed from 0 to 1, leader changed from <none> to b6c1f53ce0e346858c1fe05510cc7937 (127.2.56.1). New cstate: current_term: 1 leader_uuid: "b6c1f53ce0e346858c1fe05510cc7937" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac283b2d670b4aefb9441cf97a9fe03b" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 40817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "546ce600c3424c1a84dbb7e434d80c7a" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 33325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6c1f53ce0e346858c1fe05510cc7937" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44219 } health_report { overall_health: HEALTHY } } }
I20260928 12:22:59.100348 13984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:59.097123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56238 (local address 127.2.56.1:44219)
0928 12:22:59.097171 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 12:22:59.097173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:22:59.097183 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:22:59.097246 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:22:59.097247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:22:59.097285 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:22:59.097358 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:22:59.097567 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:59.098236 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:59.099328 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:22:59.099617 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:22:59.099680 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:22:59.099887 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:22:59.100002 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:22:59.100079 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 12:22:59.100245 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 12:22:59.100263 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:22:59.102274 13791 raft_consensus.cc:1261] T 162d8e0051e2495c9b89b69d3c8bf4d2 P 546ce600c3424c1a84dbb7e434d80c7a [term 1 FOLLOWER]: Refusing update from remote peer b6c1f53ce0e346858c1fe05510cc7937: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:22:59.102587 13925 raft_consensus.cc:1261] T 162d8e0051e2495c9b89b69d3c8bf4d2 P ac283b2d670b4aefb9441cf97a9fe03b [term 1 FOLLOWER]: Refusing update from remote peer b6c1f53ce0e346858c1fe05510cc7937: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:22:59.102897 14001 consensus_queue.cc:1035] T 162d8e0051e2495c9b89b69d3c8bf4d2 P b6c1f53ce0e346858c1fe05510cc7937 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac283b2d670b4aefb9441cf97a9fe03b" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 40817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:22:59.103047 13996 consensus_queue.cc:1035] T 162d8e0051e2495c9b89b69d3c8bf4d2 P b6c1f53ce0e346858c1fe05510cc7937 [LEADER]: Connected to new peer: Peer: permanent_uuid: "546ce600c3424c1a84dbb7e434d80c7a" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 33325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:22:59.106786 14004 mvcc.cc:205] Tried to move back new op lower bound from 7334290141600272384 to 7334290141555429376. Current Snapshot: MvccSnapshot[applied={T|T < 7334290141600272384}]
I20260928 12:22:59.107800 14007 mvcc.cc:205] Tried to move back new op lower bound from 7334290141600272384 to 7334290141555429376. Current Snapshot: MvccSnapshot[applied={T|T < 7334290141600272384 or (T in {7334290141600272384})}]
I20260928 12:22:59.108855 14005 mvcc.cc:205] Tried to move back new op lower bound from 7334290141600272384 to 7334290141555429376. Current Snapshot: MvccSnapshot[applied={T|T < 7334290141600272384}]
I20260928 12:22:59.111316 13488 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:22:59.111419 13488 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:22:59.112141 13488 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 142415bea3c348209ec5d28407ef2296: Sending DeleteTablet for 3 replicas of tablet 162d8e0051e2495c9b89b69d3c8bf4d2
I20260928 12:22:59.112457 13634 tablet_service.cc:1514] Processing DeleteTablet for tablet 162d8e0051e2495c9b89b69d3c8bf4d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:22:59 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:56234
I20260928 12:22:59.112473 13906 tablet_service.cc:1514] Processing DeleteTablet for tablet 162d8e0051e2495c9b89b69d3c8bf4d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:22:59 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:39620
I20260928 12:22:59.112828 13771 tablet_service.cc:1514] Processing DeleteTablet for tablet 162d8e0051e2495c9b89b69d3c8bf4d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:22:59 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:52164
I20260928 12:22:59.113036 14016 tablet_replica.cc:330] T 162d8e0051e2495c9b89b69d3c8bf4d2 P 546ce600c3424c1a84dbb7e434d80c7a: stopping tablet replica
I20260928 12:22:59.113045 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 13569
I20260928 12:22:59.113747 14016 raft_consensus.cc:2229] T 162d8e0051e2495c9b89b69d3c8bf4d2 P 546ce600c3424c1a84dbb7e434d80c7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:22:59.113860 14016 raft_consensus.cc:2258] T 162d8e0051e2495c9b89b69d3c8bf4d2 P 546ce600c3424c1a84dbb7e434d80c7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:22:59.114771 14016 ts_tablet_manager.cc:1905] T 162d8e0051e2495c9b89b69d3c8bf4d2 P 546ce600c3424c1a84dbb7e434d80c7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:22:59.116343 14016 ts_tablet_manager.cc:1918] T 162d8e0051e2495c9b89b69d3c8bf4d2 P 546ce600c3424c1a84dbb7e434d80c7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:22:59.116444 14016 log.cc:1196] T 162d8e0051e2495c9b89b69d3c8bf4d2 P 546ce600c3424c1a84dbb7e434d80c7a: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-1/wal/wals/162d8e0051e2495c9b89b69d3c8bf4d2
I20260928 12:22:59.117056 14016 ts_tablet_manager.cc:1939] T 162d8e0051e2495c9b89b69d3c8bf4d2 P 546ce600c3424c1a84dbb7e434d80c7a: Deleting consensus metadata
I20260928 12:22:59.117524 13474 catalog_manager.cc:4841] TS 546ce600c3424c1a84dbb7e434d80c7a (127.2.56.2:33325): tablet 162d8e0051e2495c9b89b69d3c8bf4d2 (table test-table [id=dac0a51c63ed41c582dd973694c86b82]) successfully deleted
W20260928 12:22:59.123811 13475 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.56.1:44219 (error 108)
I20260928 12:22:59.124229 14014 tablet_replica.cc:330] T 162d8e0051e2495c9b89b69d3c8bf4d2 P ac283b2d670b4aefb9441cf97a9fe03b: stopping tablet replica
I20260928 12:22:59.124366 14014 raft_consensus.cc:2229] T 162d8e0051e2495c9b89b69d3c8bf4d2 P ac283b2d670b4aefb9441cf97a9fe03b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:22:59.124500 14014 raft_consensus.cc:2258] T 162d8e0051e2495c9b89b69d3c8bf4d2 P ac283b2d670b4aefb9441cf97a9fe03b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:22:59.125049 14014 ts_tablet_manager.cc:1905] T 162d8e0051e2495c9b89b69d3c8bf4d2 P ac283b2d670b4aefb9441cf97a9fe03b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:22:59.125675 13987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:22:59.125353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:44219 (local address 127.0.0.1:56248)
0928 12:22:59.125525 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 12:22:59.125614 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.1:44219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260928 12:22:59.126119 13475 catalog_manager.cc:4584] TS b6c1f53ce0e346858c1fe05510cc7937 (127.2.56.1:44219): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 162d8e0051e2495c9b89b69d3c8bf4d2: Network error: Client connection negotiation failed: client connection to 127.2.56.1:44219: connect: Connection refused (error 111)
I20260928 12:22:59.127720 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 13706
I20260928 12:22:59.131202 14014 ts_tablet_manager.cc:1918] T 162d8e0051e2495c9b89b69d3c8bf4d2 P ac283b2d670b4aefb9441cf97a9fe03b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:22:59.131312 14014 log.cc:1196] T 162d8e0051e2495c9b89b69d3c8bf4d2 P ac283b2d670b4aefb9441cf97a9fe03b: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598072638673-2272-0/minicluster-data/ts-2/wal/wals/162d8e0051e2495c9b89b69d3c8bf4d2
I20260928 12:22:59.132025 14014 ts_tablet_manager.cc:1939] T 162d8e0051e2495c9b89b69d3c8bf4d2 P ac283b2d670b4aefb9441cf97a9fe03b: Deleting consensus metadata
I20260928 12:22:59.132563 13472 catalog_manager.cc:4841] TS ac283b2d670b4aefb9441cf97a9fe03b (127.2.56.3:40817): tablet 162d8e0051e2495c9b89b69d3c8bf4d2 (table test-table [id=dac0a51c63ed41c582dd973694c86b82]) successfully deleted
I20260928 12:22:59.135947 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 13843
I20260928 12:22:59.146525 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 13456
I20260928 12:22:59.150890 2272 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 12:22:59.484557 2272 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 12:22:59.484652 2272 mini_ranger.cc:66] Stopping Ranger...
I20260928 12:22:59.818863 2272 mini_ranger.cc:68] Stopped Ranger
2026-09-28 12:22:59.819 UTC [13042] LOG: received smart shutdown request
2026-09-28 12:22:59.822 UTC [13042] LOG: background worker "logical replication launcher" (PID 13051) exited with exit code 1
2026-09-28 12:22:59.822 UTC [13046] LOG: shutting down
2026-09-28 12:22:59.985 UTC [13042] LOG: database system is shut down
2026-09-28T12:22:59Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47659 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:23:00 dist-test-slave-920p krb5kdc[14026](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:23:00 dist-test-slave-920p krb5kdc[14026](info): set up 2 sockets
Sep 28 12:23:00 dist-test-slave-920p krb5kdc[14026](info): commencing operation
krb5kdc: starting...
W20260928 12:23:01.032974 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:23:01 dist-test-slave-920p krb5kdc[14026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:23:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:23:01Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.56.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.56.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.56.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/rangeradmin_127.2.56.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.56.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/rangeradmin_127.2.56.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.56.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.56.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.56.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/rangerlookup_127.2.56.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.56.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/rangerlookup_127.2.56.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.56.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.56.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.56.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/HTTP_127.2.56.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.56.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/HTTP_127.2.56.20@KRBTEST.COM.keytab.
I20260928 12:23:01.183174 2272 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskkXRk4O/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/postgres -l logfile start
2026-09-28 12:23:03.506 UTC [14056] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 12:23:03.506 UTC [14056] LOG: listening on IPv4 address "127.2.56.20", port 43627
2026-09-28 12:23:03.510 UTC [14056] LOG: listening on Unix socket "/tmp/.s.PGSQL.43627"
2026-09-28 12:23:03.539 UTC [14059] LOG: database system was shut down at 2026-09-28 12:23:02 UTC
2026-09-28 12:23:03.543 UTC [14056] LOG: database system is ready to accept connections
I20260928 12:23:04.404908 2272 mini_postgres.cc:96] Postgres bound to 43627
2026-09-28 12:23:04.410 UTC [14067] FATAL: database "slave" does not exist
127.2.56.20:43627 - accepting connections
I20260928 12:23:04.410713 2272 mini_ranger.cc:161] Starting Ranger...
I20260928 12:23:04.423810 2272 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 12:23:04.868963 2272 mini_ranger.cc:87] Created ranger Postgres database
I20260928 12:23:04.869041 2272 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ranger-admin
2026-09-28 12:23:05,147 [I] DB FLAVOR :POSTGRES
2026-09-28 12:23:05,147 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 12:23:05,147 [I] Checking connection..
2026-09-28 12:23:05,147 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:43627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 12:23:05,380 [I] Checking connection passed.
2026-09-28 12:23:05,380 [I] --------- Verifying version history table ---------
2026-09-28 12:23:05,380 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:43627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 12:23:05,634 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 12:23:05,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:43627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 12:23:05,861 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 12:23:05,861 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 12:23:05,861 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:43627/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 12:23:06.085 UTC [14131] WARNING: there is no transaction in progress
2026-09-28 12:23:06,091 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 12:23:06,091 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:43627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 12:23:06,335 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 12:23:06,336 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 12:23:06,336 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:43627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 12:23:06,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:43627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 12:23:06,838 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:43627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-920p.c.gcp-upstream.internal','N') ;"
2026-09-28 12:23:07,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:43627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 12:23:07,305 [I] Table x_portal_user does not exist in database ranger
2026-09-28 12:23:07,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:43627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 12:23:07,563 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 12:23:07,563 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:43627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 12:23:07,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:43627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 12:23:08,045 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 12:23:08,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:43627/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 12:23:09.082 UTC [14239] WARNING: there is no transaction in progress
2026-09-28 12:23:09.102 UTC [14239] WARNING: there is no transaction in progress
2026-09-28 12:23:09.123 UTC [14239] WARNING: there is no transaction in progress
2026-09-28 12:23:09.144 UTC [14239] WARNING: there is no transaction in progress
2026-09-28 12:23:09.164 UTC [14239] WARNING: there is no transaction in progress
2026-09-28 12:23:09.186 UTC [14239] WARNING: there is no transaction in progress
2026-09-28 12:23:09.256 UTC [14239] WARNING: there is no transaction in progress
2026-09-28 12:23:09.263 UTC [14239] WARNING: there is no transaction in progress
2026-09-28 12:23:09.278 UTC [14239] WARNING: there is no transaction in progress
2026-09-28 12:23:09.288 UTC [14239] WARNING: there is no transaction in progress
2026-09-28 12:23:09.299 UTC [14239] WARNING: there is no transaction in progress
2026-09-28 12:23:09.314 UTC [14239] WARNING: there is no transaction in progress
2026-09-28 12:23:09.326 UTC [14239] WARNING: there is no transaction in progress
2026-09-28 12:23:09.335 UTC [14239] WARNING: there is no transaction in progress
2026-09-28 12:23:09.345 UTC [14239] WARNING: there is no transaction in progress
2026-09-28 12:23:09.354 UTC [14239] WARNING: there is no transaction in progress
2026-09-28 12:23:09,992 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 12:23:09,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:43627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 12:23:10,239 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 12:23:10,240 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:43627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 12:23:10,539 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 12:23:10,539 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:43627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 12:23:10,773 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 12:23:10,774 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:43627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 12:23:11,001 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:43627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 12:23:11,247 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:43627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-920p.c.gcp-upstream.internal';"
2026-09-28 12:23:11,506 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 12:23:11,506 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:43627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 12:23:11,758 [I] DB_PATCHES have already been applied
I20260928 12:23:11.764075 2272 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ranger-admin:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskkXRk4O/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 12:23:11.764133 2272 mini_ranger.cc:193] Using host: 127.2.56.20
I20260928 12:23:11.766875 2272 mini_ranger.cc:227] Ranger admin URL: http://127.2.56.20:33569
Sep 28, 2026 12:23:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 12:23:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 12:23:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 12:23:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 12:23:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 12:23:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 12:23:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 12:23:12.750591 2272 mini_ranger.cc:160] Time spent starting Ranger: real 8.340s user 0.000s sys 0.007s
Sep 28, 2026 12:23:32 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 12:23:32 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 12:23:32 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 12:23:32 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 12:23:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 12:23:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 12:23:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 12:23:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 12:23:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 12:23:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 12:23:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 12:23:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 12:23:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 12:23:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 12:23:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 12:23:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 12:23:32 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 12:23:33 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 12:23:34.366739 2272 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.56.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.56.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.56.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/rangerkms_127.2.56.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.56.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/rangerkms_127.2.56.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.56.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.56.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.56.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/HTTP_127.2.56.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.56.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/HTTP_127.2.56.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 12:23:34 dist-test-slave-920p krb5kdc[14026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598214, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 12:23:34.444900 2272 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 12:23:34.453409 2272 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 12:23:35.315608 2272 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 12:23:35.315697 2272 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ranger-kms
I20260928 12:23:35.315716 2272 mini_ranger_kms.cc:227] Using postgres at 127.2.56.20:43627
2026-09-28 12:23:35,468 [I] DB FLAVOR :POSTGRES
2026-09-28 12:23:35,468 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 12:23:35,468 [I] Checking connection
2026-09-28 12:23:35,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:43627/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 12:23:35,709 [I] connection success
2026-09-28 12:23:35,710 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 12:23:35,710 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 12:23:35,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:43627/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 12:23:35,959 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 12:23:35,959 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 12:23:35,959 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 12:23:35,959 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.56.20:43627/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 12:23:36,205 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 12:23:36.695474 2272 mini_ranger_kms.cc:310] Created kms service
I20260928 12:23:36.904907 2272 mini_ranger_kms.cc:326] Created kudu user
I20260928 12:23:36.947175 2272 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 12:23:36 dist-test-slave-920p krb5kdc[14026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598214, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.56.20@KRBTEST.COM
I20260928 12:23:37.385689 2272 mini_ranger_kms.cc:384] Added ranger policy
I20260928 12:23:37.385953 2272 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ranger-kms:/tmp/dist-test-taskkXRk4O/build/release/bin/postgresql.jar:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskkXRk4O/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 12:23:37.385978 2272 mini_ranger_kms.cc:240] Using host: 127.2.56.20
I20260928 12:23:37.390352 2272 mini_ranger_kms.cc:276] Ranger KMS PID: 14419
I20260928 12:23:37.390455 2272 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.56.20:46899
Sep 28, 2026 12:23:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 12:23:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 12:23:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 12:23:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 12:23:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 12:23:38 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 12:23:39.500411 2272 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.056s user 0.006s sys 0.126s
I20260928 12:23:39.500504 2272 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 12:23:39.500550 2272 mini_ranger_kms.cc:401] 127.2.56.20:46899/kms/v1/keys
Sep 28 12:23:39 dist-test-slave-920p krb5kdc[14026](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790598219, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.56.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 12:23:40 dist-test-slave-920p krb5kdc[14026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598219, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.56.20@KRBTEST.COM for HTTP/127.2.56.20@KRBTEST.COM
Sep 28 12:23:40 dist-test-slave-920p krb5kdc[14026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598219, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.56.20@KRBTEST.COM for HTTP/127.2.56.20@KRBTEST.COM
Sep 28 12:23:42 dist-test-slave-920p krb5kdc[14026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
I20260928 12:23:42.765892 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:43657
--webserver_interface=127.2.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:41845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:43657
--ranger_config_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.56.20:46899/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:23:42.844900 14470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:23:42.845111 14470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:23:42.845140 14470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:23:42.846302 14470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:23:42.846344 14470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:23:42.846362 14470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:23:42.846377 14470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:23:42.846393 14470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:23:42.847903 14470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:41845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.56.20:46899/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:43657
--ranger_config_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:43657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:23:42.848129 14470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:23:42.848320 14470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:23:42.852725 14478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:23:42.852815 14470 server_base.cc:1034] running on GCE node
W20260928 12:23:42.852772 14476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:23:42.852787 14475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:23:42.853295 14470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:23:42.853621 14470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:23:42.854768 14470 hybrid_clock.cc:648] HybridClock initialized: now 1790598222854753 us; error 29 us; skew 500 ppm
Sep 28 12:23:42 dist-test-slave-920p krb5kdc[14026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598222, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:23:42.857128 14470 init.cc:377] Logged in from keytab as kudu/127.2.56.62@KRBTEST.COM (short username kudu)
I20260928 12:23:42.857561 14470 webserver.cc:460] Webserver started at http://127.2.56.62:37279/ using document root <none> and password file <none>
I20260928 12:23:42.857764 14470 fs_manager.cc:359] Metadata directory not provided
I20260928 12:23:42.857802 14470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:23:42.857918 14470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 12:23:42 dist-test-slave-920p krb5kdc[14026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598222, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for HTTP/127.2.56.20@KRBTEST.COM
I20260928 12:23:42.909704 14470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "47feda07ac204242bb669b26dd56bdc6"
format_stamp: "Formatted at 2026-09-28 12:23:42 on dist-test-slave-920p"
server_key: "d5f29c63634359b719803f78c7a003a2"
server_key_iv: "2c62dd937c8cfd27524932e855e56f20"
server_key_version: "kuduclusterkey@0"
I20260928 12:23:42.910190 14470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "47feda07ac204242bb669b26dd56bdc6"
format_stamp: "Formatted at 2026-09-28 12:23:42 on dist-test-slave-920p"
server_key: "d5f29c63634359b719803f78c7a003a2"
server_key_iv: "2c62dd937c8cfd27524932e855e56f20"
server_key_version: "kuduclusterkey@0"
I20260928 12:23:42.912146 14470 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260928 12:23:42.927569 14485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:42.927759 14470 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 12:23:42.927927 14470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "47feda07ac204242bb669b26dd56bdc6"
format_stamp: "Formatted at 2026-09-28 12:23:42 on dist-test-slave-920p"
server_key: "d5f29c63634359b719803f78c7a003a2"
server_key_iv: "2c62dd937c8cfd27524932e855e56f20"
server_key_version: "kuduclusterkey@0"
I20260928 12:23:42.928076 14470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:23:42.936700 14470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:23:42.938560 14470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:23:42.938699 14470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:23:42.943029 14470 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:43657
I20260928 12:23:42.943056 14537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:43657 every 8 connection(s)
I20260928 12:23:42.943462 14470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:23:42.944095 14538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:23:42.946794 14538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6: Bootstrap starting.
I20260928 12:23:42.947378 14538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6: Neither blocks nor log segments found. Creating new log.
I20260928 12:23:42.947638 14538 log.cc:824] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6: Log is configured to *not* fsync() on all Append() calls
I20260928 12:23:42.948315 14538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6: No bootstrap required, opened a new log
I20260928 12:23:42.950336 14538 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47feda07ac204242bb669b26dd56bdc6" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43657 } }
I20260928 12:23:42.950484 14538 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:23:42.950505 14538 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47feda07ac204242bb669b26dd56bdc6, State: Initialized, Role: FOLLOWER
I20260928 12:23:42.950634 14538 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47feda07ac204242bb669b26dd56bdc6" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43657 } }
I20260928 12:23:42.950701 14538 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:23:42.950726 14538 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:23:42.950769 14538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:23:42.951284 14538 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47feda07ac204242bb669b26dd56bdc6" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43657 } }
I20260928 12:23:42.951362 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 14470
I20260928 12:23:42.951380 14538 leader_election.cc:302] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47feda07ac204242bb669b26dd56bdc6; no voters:
I20260928 12:23:42.951459 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:23:42.951529 14538 leader_election.cc:288] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:23:42.951566 14541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:23:42.951639 2272 external_mini_cluster.cc:1428] Setting key ffd8b6494969739d33aa1552ed8a2988
I20260928 12:23:42.951731 14541 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6 [term 1 LEADER]: Becoming Leader. State: Replica: 47feda07ac204242bb669b26dd56bdc6, State: Running, Role: LEADER
I20260928 12:23:42.951803 14541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47feda07ac204242bb669b26dd56bdc6" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43657 } }
I20260928 12:23:42.951839 14538 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:23:42.952086 14541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47feda07ac204242bb669b26dd56bdc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47feda07ac204242bb669b26dd56bdc6" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43657 } } }
I20260928 12:23:42.952139 14541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6 [sys.catalog]: This master's current role is: LEADER
I20260928 12:23:42.952252 14541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47feda07ac204242bb669b26dd56bdc6. Latest consensus state: current_term: 1 leader_uuid: "47feda07ac204242bb669b26dd56bdc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47feda07ac204242bb669b26dd56bdc6" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43657 } } }
I20260928 12:23:42.952293 14541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6 [sys.catalog]: This master's current role is: LEADER
I20260928 12:23:42.953120 14538 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 12:23:42.956331 14546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:23:42.957163 14546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:23:42.959175 14546 catalog_manager.cc:1347] Generated new cluster ID: 23aa0900050449cba836715cb20aee14
I20260928 12:23:42.959275 14546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:23:42 dist-test-slave-920p krb5kdc[14026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:23:42.966713 14549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:42.952274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53126 (local address 127.2.56.62:43657)
0928 12:23:42.952496 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:42.952506 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:42.953483 (+ 977us) server_negotiation.cc:407] Connection header received
0928 12:23:42.955047 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:42.955128 (+ 81us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:42.955309 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:42.955460 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:23:42.956484 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:42.958279 (+ 1795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:42.959325 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:42.959910 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:42.959986 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:42.963000 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:23:42.963004 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:23:42.963005 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:23:42.963017 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:23:42.964938 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:42.965364 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:42.965368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:42.965368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:42.965467 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:42.965664 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:42.965664 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:42.965665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:42.965972 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:23:42.966104 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:42.966370 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 12:23:42.966502 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":63,"threads_started":1}
I20260928 12:23:42.970760 14546 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:23:42.971546 14546 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:23:42.976183 14546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6: Generated new TSK 0
I20260928 12:23:42.976437 14546 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab.
Entry for principal kudu/127.2.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab.
I20260928 12:23:43.660818 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.56.20:46899/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.1:0
--local_ip_for_outbound_sockets=127.2.56.1
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:43657
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:41845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:23:43.778679 14583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:23:43.778970 14583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:23:43.779007 14583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:23:43.780682 14583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:23:43.780743 14583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:23:43.780781 14583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.1
I20260928 12:23:43.783114 14583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:41845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.56.20:46899/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:43657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.1
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:23:43.783404 14583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:23:43.783676 14583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:23:43.789013 14588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:23:43.789012 14589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:23:43.789047 14591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:23:43.789294 14583 server_base.cc:1034] running on GCE node
I20260928 12:23:43.789605 14583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:23:43.789857 14583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:23:43.791021 14583 hybrid_clock.cc:648] HybridClock initialized: now 1790598223790995 us; error 39 us; skew 500 ppm
Sep 28 12:23:43 dist-test-slave-920p krb5kdc[14026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598223, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:23:43.794279 14583 init.cc:377] Logged in from keytab as kudu/127.2.56.1@KRBTEST.COM (short username kudu)
I20260928 12:23:43.794795 14583 webserver.cc:460] Webserver started at http://127.2.56.1:39725/ using document root <none> and password file <none>
I20260928 12:23:43.795073 14583 fs_manager.cc:359] Metadata directory not provided
I20260928 12:23:43.795132 14583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:23:43.795269 14583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 12:23:43 dist-test-slave-920p krb5kdc[14026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598223, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for HTTP/127.2.56.20@KRBTEST.COM
I20260928 12:23:43.810158 14583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-0/data/instance:
uuid: "7ab389dd4f6f49ec9ce84efe9758567a"
format_stamp: "Formatted at 2026-09-28 12:23:43 on dist-test-slave-920p"
server_key: "76dc5b5a0e3cba7c88d04ef6335e4141"
server_key_iv: "d9200a38a4a47a8e08f12f25dc0255cf"
server_key_version: "kuduclusterkey@0"
I20260928 12:23:43.810684 14583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance:
uuid: "7ab389dd4f6f49ec9ce84efe9758567a"
format_stamp: "Formatted at 2026-09-28 12:23:43 on dist-test-slave-920p"
server_key: "76dc5b5a0e3cba7c88d04ef6335e4141"
server_key_iv: "d9200a38a4a47a8e08f12f25dc0255cf"
server_key_version: "kuduclusterkey@0"
I20260928 12:23:43.812613 14583 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 12:23:43.826318 14598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:43.826529 14583 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:23:43.826619 14583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-0/wal
uuid: "7ab389dd4f6f49ec9ce84efe9758567a"
format_stamp: "Formatted at 2026-09-28 12:23:43 on dist-test-slave-920p"
server_key: "76dc5b5a0e3cba7c88d04ef6335e4141"
server_key_iv: "d9200a38a4a47a8e08f12f25dc0255cf"
server_key_version: "kuduclusterkey@0"
I20260928 12:23:43.826715 14583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:23:43.862051 14583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:23:43.864565 14583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:23:43.864719 14583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:23:43.864951 14583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:23:43.865396 14583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:23:43.865449 14583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:43.865478 14583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:23:43.865526 14583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:43.872539 14583 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.1:44787
I20260928 12:23:43.872972 14583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
I20260928 12:23:43.874737 14713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.1:44787 every 8 connection(s)
Sep 28 12:23:43 dist-test-slave-920p krb5kdc[14026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598223, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:23:43.882440 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 14583
I20260928 12:23:43.882542 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance
I20260928 12:23:43.882715 2272 external_mini_cluster.cc:1428] Setting key 5cf6717024169056a2fa64dc19746b6b
I20260928 12:23:43.883801 14716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:43.873767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43657 (local address 127.2.56.1:49249)
0928 12:23:43.873960 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:43.874004 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:43.874537 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:43.875189 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:43.875194 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:43.875392 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:23:43.875592 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:43.875596 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:43.876302 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:43.876303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:43.877066 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:43.877068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:43.877505 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:43.877506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:43.877639 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:43.878262 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:23:43.878281 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:23:43.879926 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:23:43.881965 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:43.881972 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:43.881975 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:43.882319 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:43.882577 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:43.882578 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:43.882578 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:43.882741 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:43.883084 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:23:43.883090 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:23:43.883285 (+ 195us) client_negotiation.cc:769] Sending connection context
0928 12:23:43.883403 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 12:23:43.883564 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":77,"threads_started":1}
I20260928 12:23:43.884536 14717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:43.874067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:49249 (local address 127.2.56.62:43657)
0928 12:23:43.874984 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:43.874986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:43.874994 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:23:43.875016 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:43.875017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:43.875027 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:43.875089 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:23:43.875700 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:43.876208 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:43.877155 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:43.877413 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:43.877465 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:43.880039 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:23:43.880041 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:23:43.880041 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:23:43.880049 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:23:43.881873 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:43.882414 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:43.882415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:43.882415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:43.882484 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:43.882829 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:43.882829 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:43.882830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:43.882977 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:23:43.883085 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:43.884348 (+ 1263us) server_negotiation.cc:299] Negotiation successful
0928 12:23:43.884438 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":888,"thread_start_us":75,"threads_started":1}
I20260928 12:23:43.885064 14714 heartbeater.cc:347] Connected to a master server at 127.2.56.62:43657
I20260928 12:23:43.885191 14714 heartbeater.cc:464] Registering TS with master...
I20260928 12:23:43.885427 14714 heartbeater.cc:511] Master 127.2.56.62:43657 requested a full tablet report, sending...
I20260928 12:23:43.885985 14502 ts_manager.cc:194] Registered new tserver with Master: 7ab389dd4f6f49ec9ce84efe9758567a (127.2.56.1:44787)
I20260928 12:23:43.886902 14502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.1@KRBTEST.COM'} at 127.2.56.1:49249
WARNING: no policy specified for kudu/127.2.56.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab.
Entry for principal kudu/127.2.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab.
I20260928 12:23:43.913496 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.56.20:46899/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.2:0
--local_ip_for_outbound_sockets=127.2.56.2
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:43657
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:41845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 12:23:44 dist-test-slave-920p krb5kdc[14026](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790598224, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 12:23:44.031144 14720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:23:44.031423 14720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:23:44.031466 14720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:23:44.033330 14720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:23:44.033386 14720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:23:44.033427 14720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.2
I20260928 12:23:44.036304 14720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:41845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.56.20:46899/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:43657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.2
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:23:44.036746 14720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:23:44.037072 14720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:23:44.043516 14730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:23:44.043759 14728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:23:44.043889 14727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:23:44.045436 14720 server_base.cc:1034] running on GCE node
I20260928 12:23:44.045662 14720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:23:44.045946 14720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:23:44.053334 14720 hybrid_clock.cc:648] HybridClock initialized: now 1790598224053291 us; error 45 us; skew 500 ppm
Sep 28 12:23:44 dist-test-slave-920p krb5kdc[14026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598224, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:23:44.057093 14720 init.cc:377] Logged in from keytab as kudu/127.2.56.2@KRBTEST.COM (short username kudu)
I20260928 12:23:44.057729 14720 webserver.cc:460] Webserver started at http://127.2.56.2:42969/ using document root <none> and password file <none>
I20260928 12:23:44.058089 14720 fs_manager.cc:359] Metadata directory not provided
I20260928 12:23:44.058306 14720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:23:44.058494 14720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 12:23:44 dist-test-slave-920p krb5kdc[14026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598224, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for HTTP/127.2.56.20@KRBTEST.COM
I20260928 12:23:44.077723 14720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-1/data/instance:
uuid: "2be19306ea2f4530b6865458260943c1"
format_stamp: "Formatted at 2026-09-28 12:23:44 on dist-test-slave-920p"
server_key: "70671a49165e90cb107b1f60602d4a38"
server_key_iv: "5a3764eb57670d26f7e2242831f76845"
server_key_version: "kuduclusterkey@0"
I20260928 12:23:44.078246 14720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance:
uuid: "2be19306ea2f4530b6865458260943c1"
format_stamp: "Formatted at 2026-09-28 12:23:44 on dist-test-slave-920p"
server_key: "70671a49165e90cb107b1f60602d4a38"
server_key_iv: "5a3764eb57670d26f7e2242831f76845"
server_key_version: "kuduclusterkey@0"
I20260928 12:23:44.079996 14720 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:23:44.092993 14737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:44.093571 14720 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:23:44.093691 14720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-1/wal
uuid: "2be19306ea2f4530b6865458260943c1"
format_stamp: "Formatted at 2026-09-28 12:23:44 on dist-test-slave-920p"
server_key: "70671a49165e90cb107b1f60602d4a38"
server_key_iv: "5a3764eb57670d26f7e2242831f76845"
server_key_version: "kuduclusterkey@0"
I20260928 12:23:44.093788 14720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:23:44.106829 14720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:23:44.109234 14720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:23:44.109498 14720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:23:44.109798 14720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:23:44.110206 14720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:23:44.110317 14720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:44.110390 14720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:23:44.110443 14720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:44.116603 14720 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.2:42915
I20260928 12:23:44.117154 14720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
I20260928 12:23:44.118426 14850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.2:42915 every 8 connection(s)
Sep 28 12:23:44 dist-test-slave-920p krb5kdc[14026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598224, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:23:44.127228 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 14720
I20260928 12:23:44.127351 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance
I20260928 12:23:44.127558 2272 external_mini_cluster.cc:1428] Setting key 5a4d30633c74bae13a51354a4a076012
I20260928 12:23:44.131369 14717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:44.118897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:57755 (local address 127.2.56.62:43657)
0928 12:23:44.118945 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:44.118947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:44.119207 (+ 260us) server_negotiation.cc:407] Connection header received
0928 12:23:44.119817 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:44.119819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:44.119831 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:44.119871 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:23:44.120676 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:44.121730 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:44.123212 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:44.123538 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:44.123579 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:44.126117 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:23:44.126118 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:23:44.126119 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:23:44.126126 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:23:44.127745 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:44.129738 (+ 1993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:44.129741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:44.129741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:44.129855 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:44.130221 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:44.130222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:44.130222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:44.130407 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:23:44.130453 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:44.131023 (+ 570us) server_negotiation.cc:299] Negotiation successful
0928 12:23:44.131220 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:23:44.132261 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:44.118617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43657 (local address 127.2.56.2:57755)
0928 12:23:44.119052 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:44.119098 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:44.119700 (+ 602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:44.120049 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:44.120053 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:44.120302 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:23:44.120569 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:44.120575 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:44.121870 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:44.121871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:44.123116 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:44.123118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:44.123688 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:44.123689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:44.123836 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:44.124370 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:23:44.124380 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:23:44.126015 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:23:44.128173 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:44.128181 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:44.128183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:44.129567 (+ 1384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:44.129941 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:44.129942 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:44.129942 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:44.130133 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:44.130500 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:23:44.130510 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:23:44.130772 (+ 262us) client_negotiation.cc:769] Sending connection context
0928 12:23:44.131576 (+ 804us) client_negotiation.cc:240] Negotiation successful
0928 12:23:44.131818 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":79,"threads_started":1}
I20260928 12:23:44.132951 14852 heartbeater.cc:347] Connected to a master server at 127.2.56.62:43657
I20260928 12:23:44.133054 14852 heartbeater.cc:464] Registering TS with master...
I20260928 12:23:44.133256 14852 heartbeater.cc:511] Master 127.2.56.62:43657 requested a full tablet report, sending...
I20260928 12:23:44.134577 14502 ts_manager.cc:194] Registered new tserver with Master: 2be19306ea2f4530b6865458260943c1 (127.2.56.2:42915)
I20260928 12:23:44.135042 14502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.2@KRBTEST.COM'} at 127.2.56.2:57755
WARNING: no policy specified for kudu/127.2.56.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab.
Entry for principal kudu/127.2.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab.
I20260928 12:23:44.168442 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.56.20:46899/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.3:0
--local_ip_for_outbound_sockets=127.2.56.3
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:43657
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:41845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:23:44.290424 14857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:23:44.290788 14857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:23:44.290886 14857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:23:44.292701 14857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:23:44.292927 14857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:23:44.293030 14857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.3
I20260928 12:23:44.295410 14857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:41845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.56.20:46899/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:43657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.3
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:23:44.295882 14857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:23:44.296214 14857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:23:44.302853 14863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:23:44.303282 14862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:23:44.303521 14865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:23:44.304661 14857 server_base.cc:1034] running on GCE node
I20260928 12:23:44.304868 14857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:23:44.305146 14857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:23:44.306327 14857 hybrid_clock.cc:648] HybridClock initialized: now 1790598224306304 us; error 36 us; skew 500 ppm
Sep 28 12:23:44 dist-test-slave-920p krb5kdc[14026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598224, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:23:44.309628 14857 init.cc:377] Logged in from keytab as kudu/127.2.56.3@KRBTEST.COM (short username kudu)
I20260928 12:23:44.310120 14857 webserver.cc:460] Webserver started at http://127.2.56.3:42921/ using document root <none> and password file <none>
I20260928 12:23:44.310398 14857 fs_manager.cc:359] Metadata directory not provided
I20260928 12:23:44.310451 14857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:23:44.310567 14857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 12:23:44 dist-test-slave-920p krb5kdc[14026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598224, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for HTTP/127.2.56.20@KRBTEST.COM
I20260928 12:23:44.331060 14857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-2/data/instance:
uuid: "445b33f7957e4331b6dd0d75717210c7"
format_stamp: "Formatted at 2026-09-28 12:23:44 on dist-test-slave-920p"
server_key: "1175a9ebd305b12ae951df274bc719fb"
server_key_iv: "95263da934f997fb469f820a4b220de8"
server_key_version: "kuduclusterkey@0"
I20260928 12:23:44.333616 14857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance:
uuid: "445b33f7957e4331b6dd0d75717210c7"
format_stamp: "Formatted at 2026-09-28 12:23:44 on dist-test-slave-920p"
server_key: "1175a9ebd305b12ae951df274bc719fb"
server_key_iv: "95263da934f997fb469f820a4b220de8"
server_key_version: "kuduclusterkey@0"
I20260928 12:23:44.337649 14857 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.000s
I20260928 12:23:44.350042 14872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:44.350260 14857 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:23:44.350347 14857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-2/wal
uuid: "445b33f7957e4331b6dd0d75717210c7"
format_stamp: "Formatted at 2026-09-28 12:23:44 on dist-test-slave-920p"
server_key: "1175a9ebd305b12ae951df274bc719fb"
server_key_iv: "95263da934f997fb469f820a4b220de8"
server_key_version: "kuduclusterkey@0"
I20260928 12:23:44.350441 14857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:23:44.388274 14857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:23:44.390651 14857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:23:44.390857 14857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:23:44.391178 14857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:23:44.391611 14857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:23:44.391705 14857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:44.391789 14857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:23:44.391845 14857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:44.398185 14857 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.3:32865
I20260928 12:23:44.398746 14857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
I20260928 12:23:44.399443 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 14857
I20260928 12:23:44.399547 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance
I20260928 12:23:44.399744 2272 external_mini_cluster.cc:1428] Setting key 3b5f83c1f92f9b00c37bf50d61ed33d1
I20260928 12:23:44.404616 14985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.3:32865 every 8 connection(s)
Sep 28 12:23:44 dist-test-slave-920p krb5kdc[14026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598224, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:23:44.417977 14717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:44.406274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:55147 (local address 127.2.56.62:43657)
0928 12:23:44.406317 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:44.406319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:44.406508 (+ 189us) server_negotiation.cc:407] Connection header received
0928 12:23:44.407450 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:44.407450 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:44.407462 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:44.407490 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:23:44.408498 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:44.409284 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:44.411203 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:44.411501 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:44.411537 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:44.414601 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:23:44.414602 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:23:44.414603 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:23:44.414610 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:23:44.416269 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:44.416782 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:44.416784 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:44.416784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:44.416856 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:44.417151 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:44.417152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:44.417152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:44.417325 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:23:44.417375 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:44.417774 (+ 399us) server_negotiation.cc:299] Negotiation successful
0928 12:23:44.417872 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:23:44.418738 14988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:44.405978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43657 (local address 127.2.56.3:55147)
0928 12:23:44.406390 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:44.406437 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:44.407331 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:44.407671 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:44.407675 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:44.407932 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:23:44.408396 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:44.408403 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:44.409395 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:44.409396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:44.411108 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:44.411111 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:44.411629 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:44.411630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:44.411762 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:44.412850 (+ 1088us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:23:44.412861 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:23:44.414479 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:23:44.416374 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:44.416382 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:44.416384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:44.416690 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:44.416935 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:44.416936 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:44.416936 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:44.417076 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:44.417422 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:23:44.417428 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:23:44.417614 (+ 186us) client_negotiation.cc:769] Sending connection context
0928 12:23:44.418329 (+ 715us) client_negotiation.cc:240] Negotiation successful
0928 12:23:44.418497 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":93,"threads_started":1}
I20260928 12:23:44.419483 14986 heartbeater.cc:347] Connected to a master server at 127.2.56.62:43657
I20260928 12:23:44.419584 14986 heartbeater.cc:464] Registering TS with master...
I20260928 12:23:44.419770 14986 heartbeater.cc:511] Master 127.2.56.62:43657 requested a full tablet report, sending...
I20260928 12:23:44.420214 14502 ts_manager.cc:194] Registered new tserver with Master: 445b33f7957e4331b6dd0d75717210c7 (127.2.56.3:32865)
I20260928 12:23:44.420703 14502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.3@KRBTEST.COM'} at 127.2.56.3:55147
I20260928 12:23:44.422101 2272 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:23:44.422740 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 14583
I20260928 12:23:44.429984 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 14720
I20260928 12:23:44.434930 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 14857
I20260928 12:23:44.439611 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 14470
I20260928 12:23:44.444049 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:43657
--webserver_interface=127.2.56.62
--webserver_port=37279
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:41845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:43657
--ranger_config_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.56.20:46899/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.56.20:46899/kms with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:23:44.523239 14989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:23:44.523468 14989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:23:44.523499 14989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:23:44.524677 14989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:23:44.524721 14989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:23:44.524740 14989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:23:44.524756 14989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:23:44.524773 14989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:23:44.526438 14989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:41845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.56.20:46899/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:43657
--ranger_config_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:43657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=37279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:23:44.526684 14989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:23:44.526881 14989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:23:44.531503 14994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:23:44.531654 14989 server_base.cc:1034] running on GCE node
W20260928 12:23:44.531472 14997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:23:44.531478 14995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:23:44.531999 14989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:23:44.532277 14989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:23:44.533447 14989 hybrid_clock.cc:648] HybridClock initialized: now 1790598224533428 us; error 33 us; skew 500 ppm
Sep 28 12:23:44 dist-test-slave-920p krb5kdc[14026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598224, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:23:44.535858 14989 init.cc:377] Logged in from keytab as kudu/127.2.56.62@KRBTEST.COM (short username kudu)
I20260928 12:23:44.536346 14989 webserver.cc:460] Webserver started at http://127.2.56.62:37279/ using document root <none> and password file <none>
I20260928 12:23:44.536563 14989 fs_manager.cc:359] Metadata directory not provided
I20260928 12:23:44.536612 14989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 12:23:44 dist-test-slave-920p krb5kdc[14026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598224, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for HTTP/127.2.56.20@KRBTEST.COM
I20260928 12:23:44.611797 14989 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:23:44.612526 15005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:44.612711 14989 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:23:44.612834 14989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "47feda07ac204242bb669b26dd56bdc6"
format_stamp: "Formatted at 2026-09-28 12:23:42 on dist-test-slave-920p"
server_key: "d5f29c63634359b719803f78c7a003a2"
server_key_iv: "2c62dd937c8cfd27524932e855e56f20"
server_key_version: "kuduclusterkey@0"
I20260928 12:23:44.613381 14989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:23:44.644701 14989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:23:44.647207 14989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:23:44.647389 14989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:23:44.652252 14989 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:43657
I20260928 12:23:44.652386 15057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:43657 every 8 connection(s)
I20260928 12:23:44.652686 14989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:23:44.654049 15058 sys_catalog.cc:263] Verifying existing consensus state
I20260928 12:23:44.654724 15058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6: Bootstrap starting.
I20260928 12:23:44.660156 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 14989
I20260928 12:23:44.660537 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.56.20:46899/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.1:44787
--local_ip_for_outbound_sockets=127.2.56.1
--tserver_master_addrs=127.2.56.62:43657
--webserver_port=39725
--webserver_interface=127.2.56.1
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:41845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.56.20:46899/kms with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:23:44.663358 15058 log.cc:824] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6: Log is configured to *not* fsync() on all Append() calls
I20260928 12:23:44.665191 15058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:23:44.665480 15058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6: Bootstrap complete.
I20260928 12:23:44.667732 15058 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47feda07ac204242bb669b26dd56bdc6" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43657 } }
I20260928 12:23:44.667984 15058 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47feda07ac204242bb669b26dd56bdc6, State: Initialized, Role: FOLLOWER
I20260928 12:23:44.668146 15058 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47feda07ac204242bb669b26dd56bdc6" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43657 } }
I20260928 12:23:44.668222 15058 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:23:44.668267 15058 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:23:44.668313 15058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6 [term 1 FOLLOWER]: Advancing to term 2
I20260928 12:23:44.669384 15058 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47feda07ac204242bb669b26dd56bdc6" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43657 } }
I20260928 12:23:44.669488 15058 leader_election.cc:302] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47feda07ac204242bb669b26dd56bdc6; no voters:
I20260928 12:23:44.669661 15058 leader_election.cc:288] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 12:23:44.669734 15062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 12:23:44.669880 15062 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6 [term 2 LEADER]: Becoming Leader. State: Replica: 47feda07ac204242bb669b26dd56bdc6, State: Running, Role: LEADER
I20260928 12:23:44.670049 15058 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:23:44.670084 15062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47feda07ac204242bb669b26dd56bdc6" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43657 } }
I20260928 12:23:44.670370 15064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "47feda07ac204242bb669b26dd56bdc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47feda07ac204242bb669b26dd56bdc6" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43657 } } }
I20260928 12:23:44.670413 15062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47feda07ac204242bb669b26dd56bdc6. Latest consensus state: current_term: 2 leader_uuid: "47feda07ac204242bb669b26dd56bdc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47feda07ac204242bb669b26dd56bdc6" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43657 } } }
I20260928 12:23:44.670537 15064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6 [sys.catalog]: This master's current role is: LEADER
I20260928 12:23:44.670555 15062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6 [sys.catalog]: This master's current role is: LEADER
I20260928 12:23:44.670791 15069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:23:44.671464 15069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:23:44.671638 15069 catalog_manager.cc:1259] Loaded cluster ID: 23aa0900050449cba836715cb20aee14
I20260928 12:23:44.671710 15069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:23:44.671867 15058 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 12:23:44.675065 15069 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:23:44.675650 15069 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6: Loaded TSK: 0
I20260928 12:23:44.676033 15069 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 12:23:44.778404 15060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:23:44.778790 15060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:23:44.778896 15060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:23:44.780795 15060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:23:44.780894 15060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:23:44.780968 15060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.1
I20260928 12:23:44.783438 15060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:41845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.56.20:46899/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.1:44787
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.56.1
--webserver_port=39725
--tserver_master_addrs=127.2.56.62:43657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.1
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:23:44.783844 15060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:23:44.784159 15060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:23:44.790642 15095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:23:44.790642 15096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:23:44.791115 15098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:23:44.791796 15060 server_base.cc:1034] running on GCE node
I20260928 12:23:44.792022 15060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:23:44.792312 15060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:23:44.793468 15060 hybrid_clock.cc:648] HybridClock initialized: now 1790598224793438 us; error 47 us; skew 500 ppm
Sep 28 12:23:44 dist-test-slave-920p krb5kdc[14026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598224, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:23:44.798764 15060 init.cc:377] Logged in from keytab as kudu/127.2.56.1@KRBTEST.COM (short username kudu)
I20260928 12:23:44.799391 15060 webserver.cc:460] Webserver started at http://127.2.56.1:39725/ using document root <none> and password file <none>
I20260928 12:23:44.799750 15060 fs_manager.cc:359] Metadata directory not provided
I20260928 12:23:44.799906 15060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 12:23:44 dist-test-slave-920p krb5kdc[14026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598224, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for HTTP/127.2.56.20@KRBTEST.COM
I20260928 12:23:44.826139 15060 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:23:44.827910 15106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:44.828096 15060 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 12:23:44.828197 15060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-0/wal
uuid: "7ab389dd4f6f49ec9ce84efe9758567a"
format_stamp: "Formatted at 2026-09-28 12:23:43 on dist-test-slave-920p"
server_key: "76dc5b5a0e3cba7c88d04ef6335e4141"
server_key_iv: "d9200a38a4a47a8e08f12f25dc0255cf"
server_key_version: "kuduclusterkey@0"
I20260928 12:23:44.828696 15060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:23:44.849717 15060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:23:44.852097 15060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:23:44.852313 15060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:23:44.852628 15060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:23:44.853067 15060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:23:44.853158 15060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:44.853302 15060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:23:44.853384 15060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:44.859723 15060 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.1:44787
I20260928 12:23:44.860298 15060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
I20260928 12:23:44.863004 15219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.1:44787 every 8 connection(s)
I20260928 12:23:44.866986 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 15060
I20260928 12:23:44.867290 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.56.20:46899/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.2:42915
--local_ip_for_outbound_sockets=127.2.56.2
--tserver_master_addrs=127.2.56.62:43657
--webserver_port=42969
--webserver_interface=127.2.56.2
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:41845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.56.20:46899/kms with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 12:23:44 dist-test-slave-920p krb5kdc[14026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598224, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:23:44.885967 15222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:44.863672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43657 (local address 127.2.56.1:54239)
0928 12:23:44.864040 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:44.864087 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:44.864734 (+ 647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:44.865457 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:44.865462 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:44.865695 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:23:44.865979 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:44.865985 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:44.869983 (+ 3998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:44.869984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:44.871269 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:44.871273 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:44.872281 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:44.872283 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:44.872382 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:44.872897 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:23:44.872912 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:23:44.877038 (+ 4126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:23:44.879516 (+ 2478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:44.879527 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:44.879529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:44.879896 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:44.880204 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:44.880205 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:44.880205 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:44.884675 (+ 4470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:44.885243 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:23:44.885250 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:23:44.885431 (+ 181us) client_negotiation.cc:769] Sending connection context
0928 12:23:44.885536 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 12:23:44.885710 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":115,"threads_started":1}
I20260928 12:23:44.886073 15223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:44.863714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:54239 (local address 127.2.56.62:43657)
0928 12:23:44.863964 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:44.863974 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:44.864896 (+ 922us) server_negotiation.cc:407] Connection header received
0928 12:23:44.865060 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:44.865067 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:44.865127 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:44.865259 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:23:44.866114 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:44.869833 (+ 3719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:44.871376 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:44.872042 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:44.872166 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:44.877292 (+ 5126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:23:44.877299 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:23:44.877299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:23:44.877316 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:23:44.879407 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:44.879998 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:44.880002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:44.880003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:44.880126 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:44.884779 (+ 4653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:44.884780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:44.884781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:44.885139 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:23:44.885214 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:44.885674 (+ 460us) server_negotiation.cc:299] Negotiation successful
0928 12:23:44.885835 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":70,"threads_started":1}
I20260928 12:23:44.889098 15220 heartbeater.cc:347] Connected to a master server at 127.2.56.62:43657
I20260928 12:23:44.889202 15220 heartbeater.cc:464] Registering TS with master...
I20260928 12:23:44.889412 15220 heartbeater.cc:511] Master 127.2.56.62:43657 requested a full tablet report, sending...
W20260928 12:23:44.890343 15220 heartbeater.cc:644] Failed to heartbeat to 127.2.56.62:43657 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 12:23:44.890414 15220 heartbeater.cc:464] Registering TS with master...
I20260928 12:23:44.890473 15220 heartbeater.cc:511] Master 127.2.56.62:43657 requested a full tablet report, sending...
I20260928 12:23:44.890889 15220 heartbeater.cc:464] Registering TS with master...
I20260928 12:23:44.890957 15220 heartbeater.cc:511] Master 127.2.56.62:43657 requested a full tablet report, sending...
W20260928 12:23:44.891283 15220 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 12:23:44.979177 15224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:23:44.979399 15224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:23:44.979424 15224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:23:44.980757 15224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:23:44.980808 15224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:23:44.980845 15224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.2
I20260928 12:23:44.982702 15224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:41845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.56.20:46899/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.2:42915
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.56.2
--webserver_port=42969
--tserver_master_addrs=127.2.56.62:43657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.2
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:23:44.983042 15224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:23:44.983273 15224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:23:44.988175 15229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:23:44.988557 15230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:23:44.988756 15224 server_base.cc:1034] running on GCE node
W20260928 12:23:44.989010 15232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:23:44.989248 15224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:23:44.989537 15224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:23:44.990723 15224 hybrid_clock.cc:648] HybridClock initialized: now 1790598224990710 us; error 27 us; skew 500 ppm
Sep 28 12:23:44 dist-test-slave-920p krb5kdc[14026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598224, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:23:44.997804 15224 init.cc:377] Logged in from keytab as kudu/127.2.56.2@KRBTEST.COM (short username kudu)
I20260928 12:23:44.998505 15224 webserver.cc:460] Webserver started at http://127.2.56.2:42969/ using document root <none> and password file <none>
I20260928 12:23:44.998823 15224 fs_manager.cc:359] Metadata directory not provided
I20260928 12:23:44.998974 15224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 12:23:45 dist-test-slave-920p krb5kdc[14026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598224, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for HTTP/127.2.56.20@KRBTEST.COM
I20260928 12:23:45.017908 15224 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:23:45.019496 15240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:45.019706 15224 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:23:45.019801 15224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-1/wal
uuid: "2be19306ea2f4530b6865458260943c1"
format_stamp: "Formatted at 2026-09-28 12:23:44 on dist-test-slave-920p"
server_key: "70671a49165e90cb107b1f60602d4a38"
server_key_iv: "5a3764eb57670d26f7e2242831f76845"
server_key_version: "kuduclusterkey@0"
I20260928 12:23:45.020292 15224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:23:45.048422 15224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:23:45.050375 15224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:23:45.050510 15224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:23:45.050760 15224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:23:45.051121 15224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:23:45.051157 15224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:45.051188 15224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:23:45.051211 15224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:45.058017 15224 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.2:42915
I20260928 12:23:45.058099 15353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.2:42915 every 8 connection(s)
I20260928 12:23:45.058645 15224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
Sep 28 12:23:45 dist-test-slave-920p krb5kdc[14026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598224, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:23:45.068261 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 15224
I20260928 12:23:45.068583 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.56.20:46899/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.3:32865
--local_ip_for_outbound_sockets=127.2.56.3
--tserver_master_addrs=127.2.56.62:43657
--webserver_port=42921
--webserver_interface=127.2.56.3
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:41845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.56.20:46899/kms with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:23:45.070001 15223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:45.059438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:57545 (local address 127.2.56.62:43657)
0928 12:23:45.059491 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:45.059493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:45.059765 (+ 272us) server_negotiation.cc:407] Connection header received
0928 12:23:45.060428 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:45.060430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:45.060448 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:45.060517 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:23:45.061226 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:45.062065 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:45.063589 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:45.063889 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:45.063938 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:45.066739 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:23:45.066740 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:23:45.066741 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:23:45.066748 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:23:45.068446 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:45.068926 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:45.068928 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:45.068928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:45.068986 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:45.069240 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:45.069241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:45.069241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:45.069410 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:23:45.069452 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:45.069812 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 12:23:45.069904 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 12:23:45.070762 15356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:45.059439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43657 (local address 127.2.56.2:57545)
0928 12:23:45.059607 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:45.059655 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:45.060289 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:45.060597 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:45.060602 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:45.060830 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:23:45.061111 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:45.061117 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:45.062195 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:45.062196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:45.063474 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:45.063478 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:45.064062 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:45.064063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:45.064255 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:45.064888 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:23:45.064899 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:23:45.066619 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:23:45.068536 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:45.068543 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:45.068546 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:45.068838 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:45.069049 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:45.069050 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:45.069050 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:45.069175 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:45.069497 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:23:45.069503 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:23:45.069671 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 12:23:45.070349 (+ 678us) client_negotiation.cc:240] Negotiation successful
0928 12:23:45.070528 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
I20260928 12:23:45.071437 15354 heartbeater.cc:347] Connected to a master server at 127.2.56.62:43657
I20260928 12:23:45.071537 15354 heartbeater.cc:464] Registering TS with master...
I20260928 12:23:45.071719 15354 heartbeater.cc:511] Master 127.2.56.62:43657 requested a full tablet report, sending...
W20260928 12:23:45.072371 15354 heartbeater.cc:644] Failed to heartbeat to 127.2.56.62:43657 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 12:23:45.072480 15354 heartbeater.cc:464] Registering TS with master...
I20260928 12:23:45.072570 15354 heartbeater.cc:511] Master 127.2.56.62:43657 requested a full tablet report, sending...
I20260928 12:23:45.072886 15354 heartbeater.cc:464] Registering TS with master...
I20260928 12:23:45.072999 15354 heartbeater.cc:511] Master 127.2.56.62:43657 requested a full tablet report, sending...
W20260928 12:23:45.073303 15354 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 12:23:45.181033 15357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:23:45.181224 15357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:23:45.181254 15357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:23:45.182511 15357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:23:45.182603 15357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:23:45.182669 15357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.3
I20260928 12:23:45.184331 15357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:41845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.56.20:46899/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.3:32865
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.56.3
--webserver_port=42921
--tserver_master_addrs=127.2.56.62:43657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.3
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:23:45.184561 15357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:23:45.184803 15357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:23:45.191056 15363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:23:45.191427 15364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:23:45.191779 15366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:23:45.192951 15357 server_base.cc:1034] running on GCE node
I20260928 12:23:45.193202 15357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:23:45.193580 15357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:23:45.194810 15357 hybrid_clock.cc:648] HybridClock initialized: now 1790598225194796 us; error 27 us; skew 500 ppm
Sep 28 12:23:45 dist-test-slave-920p krb5kdc[14026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598225, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:23:45.197356 15357 init.cc:377] Logged in from keytab as kudu/127.2.56.3@KRBTEST.COM (short username kudu)
I20260928 12:23:45.197806 15357 webserver.cc:460] Webserver started at http://127.2.56.3:42921/ using document root <none> and password file <none>
I20260928 12:23:45.198061 15357 fs_manager.cc:359] Metadata directory not provided
I20260928 12:23:45.198123 15357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 12:23:45 dist-test-slave-920p krb5kdc[14026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598225, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for HTTP/127.2.56.20@KRBTEST.COM
I20260928 12:23:45.216796 15357 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
I20260928 12:23:45.223748 15374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:45.223984 15357 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260928 12:23:45.224061 15357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-2/wal
uuid: "445b33f7957e4331b6dd0d75717210c7"
format_stamp: "Formatted at 2026-09-28 12:23:44 on dist-test-slave-920p"
server_key: "1175a9ebd305b12ae951df274bc719fb"
server_key_iv: "95263da934f997fb469f820a4b220de8"
server_key_version: "kuduclusterkey@0"
I20260928 12:23:45.225188 15357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:23:45.242090 15357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:23:45.244082 15357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:23:45.244279 15357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:23:45.244565 15357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:23:45.244966 15357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:23:45.245047 15357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:45.245108 15357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:23:45.245152 15357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:45.253729 15357 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.3:32865
I20260928 12:23:45.253813 15487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.3:32865 every 8 connection(s)
I20260928 12:23:45.254182 15357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
I20260928 12:23:45.260147 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 15357
Sep 28 12:23:45 dist-test-slave-920p krb5kdc[14026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598225, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:23:45.266053 15223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:45.254943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:44841 (local address 127.2.56.62:43657)
0928 12:23:45.254987 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:45.254989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:45.255255 (+ 266us) server_negotiation.cc:407] Connection header received
0928 12:23:45.255774 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:45.255776 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:45.255788 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:45.255875 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:23:45.256443 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:45.257224 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:45.258368 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:45.258678 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:45.258756 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:45.262030 (+ 3274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:23:45.262032 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:23:45.262032 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:23:45.262039 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:23:45.264530 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:45.264986 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:45.264987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:45.264987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:45.265042 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:45.265351 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:45.265352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:45.265352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:45.265471 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:23:45.265548 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:45.265878 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 12:23:45.265958 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 12:23:45.266161 15490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:45.254965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43657 (local address 127.2.56.3:44841)
0928 12:23:45.255117 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:45.255154 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:45.255664 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:45.255928 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:45.255931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:45.256124 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:23:45.256351 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:45.256356 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:45.257377 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:45.257378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:45.258278 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:45.258280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:45.258785 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:45.258786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:45.258895 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:45.260460 (+ 1565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:23:45.260469 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:23:45.261931 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:23:45.264626 (+ 2695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:45.264634 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:45.264636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:45.264921 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:45.265109 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:45.265110 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:45.265111 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:45.265246 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:45.265529 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:23:45.265535 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:23:45.265698 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 12:23:45.265844 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 12:23:45.265978 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":87,"threads_started":1}
I20260928 12:23:45.266600 15488 heartbeater.cc:347] Connected to a master server at 127.2.56.62:43657
I20260928 12:23:45.266696 15488 heartbeater.cc:464] Registering TS with master...
I20260928 12:23:45.266898 15488 heartbeater.cc:511] Master 127.2.56.62:43657 requested a full tablet report, sending...
W20260928 12:23:45.267509 15488 heartbeater.cc:644] Failed to heartbeat to 127.2.56.62:43657 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 12:23:45.267570 15488 heartbeater.cc:464] Registering TS with master...
I20260928 12:23:45.267623 15488 heartbeater.cc:511] Master 127.2.56.62:43657 requested a full tablet report, sending...
I20260928 12:23:45.267951 15488 heartbeater.cc:464] Registering TS with master...
I20260928 12:23:45.268021 15488 heartbeater.cc:511] Master 127.2.56.62:43657 requested a full tablet report, sending...
W20260928 12:23:45.268308 15488 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 12:23:45.268628 15492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:45.260705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58956 (local address 127.2.56.62:43657)
0928 12:23:45.260877 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:45.260878 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:45.260885 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:23:45.260905 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:45.260906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:45.260917 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:45.260985 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:23:45.261709 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:45.262343 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:45.263410 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:45.263598 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:45.263656 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:45.264828 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:23:45.264829 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:23:45.264829 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:23:45.264837 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:23:45.266958 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:45.267412 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:45.267413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:45.267413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:45.267470 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:45.267930 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:45.267931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:45.267931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:45.268061 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:23:45.268213 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:45.268455 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 12:23:45.268537 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":58,"threads_started":1}
Sep 28 12:23:45 dist-test-slave-920p krb5kdc[14026](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790598225, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:23:45.891453 15220 heartbeater.cc:464] Registering TS with master...
I20260928 12:23:45.891618 15220 heartbeater.cc:511] Master 127.2.56.62:43657 requested a full tablet report, sending...
I20260928 12:23:45.891990 15022 ts_manager.cc:194] Registered new tserver with Master: 7ab389dd4f6f49ec9ce84efe9758567a (127.2.56.1:44787)
I20260928 12:23:45.893028 15022 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.1@KRBTEST.COM'} at 127.2.56.1:54239
I20260928 12:23:46.073506 15354 heartbeater.cc:464] Registering TS with master...
I20260928 12:23:46.073647 15354 heartbeater.cc:511] Master 127.2.56.62:43657 requested a full tablet report, sending...
I20260928 12:23:46.074084 15022 ts_manager.cc:194] Registered new tserver with Master: 2be19306ea2f4530b6865458260943c1 (127.2.56.2:42915)
I20260928 12:23:46.074604 15022 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.2@KRBTEST.COM'} at 127.2.56.2:57545
Sep 28 12:23:46 dist-test-slave-920p krb5kdc[14026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598225, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for HTTP/127.2.56.20@KRBTEST.COM
I20260928 12:23:46.268481 15488 heartbeater.cc:464] Registering TS with master...
I20260928 12:23:46.268633 15488 heartbeater.cc:511] Master 127.2.56.62:43657 requested a full tablet report, sending...
I20260928 12:23:46.268923 15022 ts_manager.cc:194] Registered new tserver with Master: 445b33f7957e4331b6dd0d75717210c7 (127.2.56.3:32865)
I20260928 12:23:46.269146 2272 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:23:46.269464 15022 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.3@KRBTEST.COM'} at 127.2.56.3:44841
I20260928 12:23:46.283185 15510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:46.270324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58958 (local address 127.2.56.62:43657)
0928 12:23:46.270609 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:46.270611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:46.270620 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:23:46.270642 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:46.270642 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:46.270655 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:46.270770 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:23:46.271483 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:46.272221 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:46.273261 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:46.273598 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:46.273663 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:46.280210 (+ 6547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:23:46.280212 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:23:46.280212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:23:46.280221 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:23:46.281999 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:46.282369 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:46.282370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:46.282371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:46.282437 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:46.282622 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:46.282623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:46.282623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:46.282767 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:23:46.282864 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:46.282985 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 12:23:46.283074 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":52,"threads_started":1}
I20260928 12:23:46.285535 15022 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:23:46.287889 15022 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:23:46.296242 15516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:46.290820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:44787 (local address 127.0.0.1:59212)
0928 12:23:46.291349 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:46.291550 (+ 201us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:46.291595 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:46.291925 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:46.291929 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:46.291931 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:23:46.292003 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:46.292005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:46.293390 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:46.293391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:46.294784 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:46.294787 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:46.295870 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:46.295871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:46.296028 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:46.296046 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 12:23:46.296112 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 12:23:46.296160 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":87,"threads_started":1}
I20260928 12:23:46.296263 15518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:46.291155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59212 (local address 127.2.56.1:44787)
0928 12:23:46.291291 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:46.291296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:46.291648 (+ 352us) server_negotiation.cc:407] Connection header received
0928 12:23:46.291766 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:46.291769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:46.291786 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:46.291862 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:23:46.292077 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:46.293184 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:46.294878 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:46.295651 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:46.295725 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:46.295791 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:46.296137 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 12:23:46.296184 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260928 12:23:46.297590 15154 tablet_service.cc:1467] Processing CreateTablet for tablet 5368e56e23d842d290c27a3aaf5cc7ef (DEFAULT_TABLE table=test-table [id=50a558dba05a49acac8b66bf168b6ef6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:23:46.297976 15154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5368e56e23d842d290c27a3aaf5cc7ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:23:46.301013 15519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:46.290820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:32865 (local address 127.0.0.1:35616)
0928 12:23:46.291351 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:46.291390 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:46.291456 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:46.297545 (+ 6089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:46.297546 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:46.297548 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:23:46.297589 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:46.297590 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:46.298869 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:46.298870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:46.299894 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:46.299897 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:46.300770 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:46.300771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:46.300844 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:46.300848 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:23:46.300874 (+ 26us) client_negotiation.cc:240] Negotiation successful
0928 12:23:46.300917 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":2,"thread_start_us":275,"threads_started":1}
I20260928 12:23:46.301362 15517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:46.291042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35616 (local address 127.2.56.3:32865)
0928 12:23:46.297384 (+ 6342us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:46.297388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:46.297398 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:23:46.297431 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:46.297433 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:46.297448 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:46.297647 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:23:46.297729 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:46.298757 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:46.299988 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:46.300682 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:46.301121 (+ 439us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:46.301190 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:46.301215 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 12:23:46.301256 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6306,"thread_start_us":61,"threads_started":1}
I20260928 12:23:46.302047 15407 tablet_service.cc:1467] Processing CreateTablet for tablet 5368e56e23d842d290c27a3aaf5cc7ef (DEFAULT_TABLE table=test-table [id=50a558dba05a49acac8b66bf168b6ef6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:23:46.302435 15407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5368e56e23d842d290c27a3aaf5cc7ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:23:46.307104 15522 tablet_bootstrap.cc:492] T 5368e56e23d842d290c27a3aaf5cc7ef P 7ab389dd4f6f49ec9ce84efe9758567a: Bootstrap starting.
I20260928 12:23:46.307832 15522 tablet_bootstrap.cc:654] T 5368e56e23d842d290c27a3aaf5cc7ef P 7ab389dd4f6f49ec9ce84efe9758567a: Neither blocks nor log segments found. Creating new log.
I20260928 12:23:46.308202 15522 log.cc:824] T 5368e56e23d842d290c27a3aaf5cc7ef P 7ab389dd4f6f49ec9ce84efe9758567a: Log is configured to *not* fsync() on all Append() calls
I20260928 12:23:46.309427 15521 tablet_bootstrap.cc:492] T 5368e56e23d842d290c27a3aaf5cc7ef P 445b33f7957e4331b6dd0d75717210c7: Bootstrap starting.
I20260928 12:23:46.310221 15521 tablet_bootstrap.cc:654] T 5368e56e23d842d290c27a3aaf5cc7ef P 445b33f7957e4331b6dd0d75717210c7: Neither blocks nor log segments found. Creating new log.
I20260928 12:23:46.310530 15515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:46.290851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:42915 (local address 127.0.0.1:55622)
0928 12:23:46.296424 (+ 5573us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:46.296435 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:46.296483 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:46.296717 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:46.296718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:46.296720 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:23:46.296758 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:46.296759 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:46.308351 (+ 11592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:46.308352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:46.309444 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:46.309447 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:46.310318 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:46.310319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:46.310376 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:46.310380 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:23:46.310405 (+ 25us) client_negotiation.cc:240] Negotiation successful
0928 12:23:46.310445 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5550,"thread_start_us":370,"threads_started":1}
I20260928 12:23:46.310840 15520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:46.291303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55622 (local address 127.2.56.2:42915)
0928 12:23:46.296590 (+ 5287us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:46.296595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:46.296604 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:23:46.296636 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:46.296638 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:46.296653 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:46.296721 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:23:46.296830 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:46.308240 (+ 11410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:46.309542 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:46.310229 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:46.310632 (+ 403us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:46.310699 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:46.310724 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 12:23:46.310764 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5256,"thread_start_us":55,"threads_started":1}
I20260928 12:23:46.310997 15521 log.cc:824] T 5368e56e23d842d290c27a3aaf5cc7ef P 445b33f7957e4331b6dd0d75717210c7: Log is configured to *not* fsync() on all Append() calls
I20260928 12:23:46.311566 15522 tablet_bootstrap.cc:492] T 5368e56e23d842d290c27a3aaf5cc7ef P 7ab389dd4f6f49ec9ce84efe9758567a: No bootstrap required, opened a new log
I20260928 12:23:46.311498 15288 tablet_service.cc:1467] Processing CreateTablet for tablet 5368e56e23d842d290c27a3aaf5cc7ef (DEFAULT_TABLE table=test-table [id=50a558dba05a49acac8b66bf168b6ef6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:23:46.311641 15522 ts_tablet_manager.cc:1397] T 5368e56e23d842d290c27a3aaf5cc7ef P 7ab389dd4f6f49ec9ce84efe9758567a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 12:23:46.311777 15288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5368e56e23d842d290c27a3aaf5cc7ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:23:46.314642 15522 raft_consensus.cc:348] T 5368e56e23d842d290c27a3aaf5cc7ef P 7ab389dd4f6f49ec9ce84efe9758567a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445b33f7957e4331b6dd0d75717210c7" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 32865 } } peers { permanent_uuid: "7ab389dd4f6f49ec9ce84efe9758567a" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44787 } } peers { permanent_uuid: "2be19306ea2f4530b6865458260943c1" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 42915 } }
I20260928 12:23:46.314922 15522 raft_consensus.cc:374] T 5368e56e23d842d290c27a3aaf5cc7ef P 7ab389dd4f6f49ec9ce84efe9758567a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:23:46.315016 15522 raft_consensus.cc:729] T 5368e56e23d842d290c27a3aaf5cc7ef P 7ab389dd4f6f49ec9ce84efe9758567a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ab389dd4f6f49ec9ce84efe9758567a, State: Initialized, Role: FOLLOWER
I20260928 12:23:46.315220 15522 consensus_queue.cc:260] T 5368e56e23d842d290c27a3aaf5cc7ef P 7ab389dd4f6f49ec9ce84efe9758567a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445b33f7957e4331b6dd0d75717210c7" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 32865 } } peers { permanent_uuid: "7ab389dd4f6f49ec9ce84efe9758567a" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44787 } } peers { permanent_uuid: "2be19306ea2f4530b6865458260943c1" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 42915 } }
I20260928 12:23:46.315636 15522 ts_tablet_manager.cc:1428] T 5368e56e23d842d290c27a3aaf5cc7ef P 7ab389dd4f6f49ec9ce84efe9758567a: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 12:23:46.315848 15220 heartbeater.cc:503] Master 127.2.56.62:43657 was elected leader, sending a full tablet report...
I20260928 12:23:46.316355 15524 tablet_bootstrap.cc:492] T 5368e56e23d842d290c27a3aaf5cc7ef P 2be19306ea2f4530b6865458260943c1: Bootstrap starting.
I20260928 12:23:46.317000 15524 tablet_bootstrap.cc:654] T 5368e56e23d842d290c27a3aaf5cc7ef P 2be19306ea2f4530b6865458260943c1: Neither blocks nor log segments found. Creating new log.
I20260928 12:23:46.317308 15524 log.cc:824] T 5368e56e23d842d290c27a3aaf5cc7ef P 2be19306ea2f4530b6865458260943c1: Log is configured to *not* fsync() on all Append() calls
I20260928 12:23:46.317903 15521 tablet_bootstrap.cc:492] T 5368e56e23d842d290c27a3aaf5cc7ef P 445b33f7957e4331b6dd0d75717210c7: No bootstrap required, opened a new log
I20260928 12:23:46.317978 15521 ts_tablet_manager.cc:1397] T 5368e56e23d842d290c27a3aaf5cc7ef P 445b33f7957e4331b6dd0d75717210c7: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260928 12:23:46.320315 15521 raft_consensus.cc:348] T 5368e56e23d842d290c27a3aaf5cc7ef P 445b33f7957e4331b6dd0d75717210c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445b33f7957e4331b6dd0d75717210c7" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 32865 } } peers { permanent_uuid: "7ab389dd4f6f49ec9ce84efe9758567a" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44787 } } peers { permanent_uuid: "2be19306ea2f4530b6865458260943c1" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 42915 } }
I20260928 12:23:46.320487 15521 raft_consensus.cc:374] T 5368e56e23d842d290c27a3aaf5cc7ef P 445b33f7957e4331b6dd0d75717210c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:23:46.320515 15521 raft_consensus.cc:729] T 5368e56e23d842d290c27a3aaf5cc7ef P 445b33f7957e4331b6dd0d75717210c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 445b33f7957e4331b6dd0d75717210c7, State: Initialized, Role: FOLLOWER
I20260928 12:23:46.320636 15521 consensus_queue.cc:260] T 5368e56e23d842d290c27a3aaf5cc7ef P 445b33f7957e4331b6dd0d75717210c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445b33f7957e4331b6dd0d75717210c7" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 32865 } } peers { permanent_uuid: "7ab389dd4f6f49ec9ce84efe9758567a" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44787 } } peers { permanent_uuid: "2be19306ea2f4530b6865458260943c1" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 42915 } }
I20260928 12:23:46.320870 15521 ts_tablet_manager.cc:1428] T 5368e56e23d842d290c27a3aaf5cc7ef P 445b33f7957e4331b6dd0d75717210c7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:23:46.320914 15488 heartbeater.cc:503] Master 127.2.56.62:43657 was elected leader, sending a full tablet report...
I20260928 12:23:46.321812 15524 tablet_bootstrap.cc:492] T 5368e56e23d842d290c27a3aaf5cc7ef P 2be19306ea2f4530b6865458260943c1: No bootstrap required, opened a new log
I20260928 12:23:46.321918 15524 ts_tablet_manager.cc:1397] T 5368e56e23d842d290c27a3aaf5cc7ef P 2be19306ea2f4530b6865458260943c1: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260928 12:23:46.324059 15524 raft_consensus.cc:348] T 5368e56e23d842d290c27a3aaf5cc7ef P 2be19306ea2f4530b6865458260943c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445b33f7957e4331b6dd0d75717210c7" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 32865 } } peers { permanent_uuid: "7ab389dd4f6f49ec9ce84efe9758567a" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44787 } } peers { permanent_uuid: "2be19306ea2f4530b6865458260943c1" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 42915 } }
I20260928 12:23:46.324260 15524 raft_consensus.cc:374] T 5368e56e23d842d290c27a3aaf5cc7ef P 2be19306ea2f4530b6865458260943c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:23:46.324369 15524 raft_consensus.cc:729] T 5368e56e23d842d290c27a3aaf5cc7ef P 2be19306ea2f4530b6865458260943c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2be19306ea2f4530b6865458260943c1, State: Initialized, Role: FOLLOWER
I20260928 12:23:46.324484 15524 consensus_queue.cc:260] T 5368e56e23d842d290c27a3aaf5cc7ef P 2be19306ea2f4530b6865458260943c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445b33f7957e4331b6dd0d75717210c7" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 32865 } } peers { permanent_uuid: "7ab389dd4f6f49ec9ce84efe9758567a" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44787 } } peers { permanent_uuid: "2be19306ea2f4530b6865458260943c1" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 42915 } }
I20260928 12:23:46.324708 15524 ts_tablet_manager.cc:1428] T 5368e56e23d842d290c27a3aaf5cc7ef P 2be19306ea2f4530b6865458260943c1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:23:46.324873 15354 heartbeater.cc:503] Master 127.2.56.62:43657 was elected leader, sending a full tablet report...
W20260928 12:23:46.364771 15221 tablet.cc:2369] T 5368e56e23d842d290c27a3aaf5cc7ef P 7ab389dd4f6f49ec9ce84efe9758567a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 12:23:46 dist-test-slave-920p krb5kdc[14026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598225, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for HTTP/127.2.56.20@KRBTEST.COM
W20260928 12:23:46.505674 15489 tablet.cc:2369] T 5368e56e23d842d290c27a3aaf5cc7ef P 445b33f7957e4331b6dd0d75717210c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:23:46.560132 15355 tablet.cc:2369] T 5368e56e23d842d290c27a3aaf5cc7ef P 2be19306ea2f4530b6865458260943c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:23:46.622349 15529 raft_consensus.cc:482] T 5368e56e23d842d290c27a3aaf5cc7ef P 2be19306ea2f4530b6865458260943c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:23:46.622490 15529 raft_consensus.cc:504] T 5368e56e23d842d290c27a3aaf5cc7ef P 2be19306ea2f4530b6865458260943c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445b33f7957e4331b6dd0d75717210c7" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 32865 } } peers { permanent_uuid: "7ab389dd4f6f49ec9ce84efe9758567a" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44787 } } peers { permanent_uuid: "2be19306ea2f4530b6865458260943c1" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 42915 } }
I20260928 12:23:46.622885 15529 leader_election.cc:288] T 5368e56e23d842d290c27a3aaf5cc7ef P 2be19306ea2f4530b6865458260943c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 445b33f7957e4331b6dd0d75717210c7 (127.2.56.3:32865), 7ab389dd4f6f49ec9ce84efe9758567a (127.2.56.1:44787)
I20260928 12:23:46.627141 15517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:46.623259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:58163 (local address 127.2.56.3:32865)
0928 12:23:46.623367 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:46.623369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:46.623513 (+ 144us) server_negotiation.cc:407] Connection header received
0928 12:23:46.623598 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:46.623598 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:46.623609 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:46.623636 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:23:46.623855 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:46.624580 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:46.626145 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:46.626736 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:46.626781 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:46.626791 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:46.627027 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 12:23:46.627067 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 12:23:46.627141 15518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:46.623311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:37255 (local address 127.2.56.1:44787)
0928 12:23:46.623405 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:46.623407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:46.623415 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:23:46.623444 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:46.623445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:46.623456 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:46.623505 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:23:46.623736 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:46.624547 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:46.626145 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:46.626736 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:46.626781 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:46.626791 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:46.627027 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 12:23:46.627067 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 12:23:46.627417 15533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:46.623055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:44787 (local address 127.2.56.2:37255)
0928 12:23:46.623304 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:46.623327 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:46.623390 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:46.623581 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:46.623582 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:46.623584 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:23:46.623660 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:46.623663 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:46.624677 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:46.624678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:46.626037 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:46.626040 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:46.626873 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:46.626874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:46.626945 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:46.626950 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:23:46.627261 (+ 311us) client_negotiation.cc:240] Negotiation successful
0928 12:23:46.627310 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":46,"thread_start_us":95,"threads_started":1}
I20260928 12:23:46.627664 15532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:46.623055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:32865 (local address 127.2.56.2:58163)
0928 12:23:46.623466 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:46.623474 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:46.623532 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:46.623690 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:46.623690 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:46.623692 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:23:46.623764 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:46.623766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:46.624677 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:46.624678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:46.626037 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:46.626040 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:46.626873 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:46.626874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:46.626948 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:46.626950 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:23:46.627557 (+ 607us) client_negotiation.cc:240] Negotiation successful
0928 12:23:46.627593 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":114,"thread_start_us":84,"threads_started":1}
I20260928 12:23:46.627785 15172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5368e56e23d842d290c27a3aaf5cc7ef" candidate_uuid: "2be19306ea2f4530b6865458260943c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ab389dd4f6f49ec9ce84efe9758567a" is_pre_election: true
I20260928 12:23:46.627956 15172 raft_consensus.cc:2454] T 5368e56e23d842d290c27a3aaf5cc7ef P 7ab389dd4f6f49ec9ce84efe9758567a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2be19306ea2f4530b6865458260943c1 in term 0.
I20260928 12:23:46.628134 15426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5368e56e23d842d290c27a3aaf5cc7ef" candidate_uuid: "2be19306ea2f4530b6865458260943c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "445b33f7957e4331b6dd0d75717210c7" is_pre_election: true
I20260928 12:23:46.628166 15242 leader_election.cc:302] T 5368e56e23d842d290c27a3aaf5cc7ef P 2be19306ea2f4530b6865458260943c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2be19306ea2f4530b6865458260943c1, 7ab389dd4f6f49ec9ce84efe9758567a; no voters:
I20260928 12:23:46.628261 15426 raft_consensus.cc:2454] T 5368e56e23d842d290c27a3aaf5cc7ef P 445b33f7957e4331b6dd0d75717210c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2be19306ea2f4530b6865458260943c1 in term 0.
I20260928 12:23:46.628283 15529 raft_consensus.cc:2781] T 5368e56e23d842d290c27a3aaf5cc7ef P 2be19306ea2f4530b6865458260943c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:23:46.628319 15529 raft_consensus.cc:482] T 5368e56e23d842d290c27a3aaf5cc7ef P 2be19306ea2f4530b6865458260943c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:23:46.628353 15529 raft_consensus.cc:3037] T 5368e56e23d842d290c27a3aaf5cc7ef P 2be19306ea2f4530b6865458260943c1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:23:46.629211 15529 raft_consensus.cc:504] T 5368e56e23d842d290c27a3aaf5cc7ef P 2be19306ea2f4530b6865458260943c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445b33f7957e4331b6dd0d75717210c7" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 32865 } } peers { permanent_uuid: "7ab389dd4f6f49ec9ce84efe9758567a" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44787 } } peers { permanent_uuid: "2be19306ea2f4530b6865458260943c1" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 42915 } }
I20260928 12:23:46.629387 15529 leader_election.cc:288] T 5368e56e23d842d290c27a3aaf5cc7ef P 2be19306ea2f4530b6865458260943c1 [CANDIDATE]: Term 1 election: Requested vote from peers 445b33f7957e4331b6dd0d75717210c7 (127.2.56.3:32865), 7ab389dd4f6f49ec9ce84efe9758567a (127.2.56.1:44787)
I20260928 12:23:46.629571 15426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5368e56e23d842d290c27a3aaf5cc7ef" candidate_uuid: "2be19306ea2f4530b6865458260943c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "445b33f7957e4331b6dd0d75717210c7"
I20260928 12:23:46.629611 15172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5368e56e23d842d290c27a3aaf5cc7ef" candidate_uuid: "2be19306ea2f4530b6865458260943c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ab389dd4f6f49ec9ce84efe9758567a"
I20260928 12:23:46.629650 15426 raft_consensus.cc:3037] T 5368e56e23d842d290c27a3aaf5cc7ef P 445b33f7957e4331b6dd0d75717210c7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:23:46.629673 15172 raft_consensus.cc:3037] T 5368e56e23d842d290c27a3aaf5cc7ef P 7ab389dd4f6f49ec9ce84efe9758567a [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:23:46.630463 15426 raft_consensus.cc:2454] T 5368e56e23d842d290c27a3aaf5cc7ef P 445b33f7957e4331b6dd0d75717210c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be19306ea2f4530b6865458260943c1 in term 1.
I20260928 12:23:46.630463 15172 raft_consensus.cc:2454] T 5368e56e23d842d290c27a3aaf5cc7ef P 7ab389dd4f6f49ec9ce84efe9758567a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be19306ea2f4530b6865458260943c1 in term 1.
I20260928 12:23:46.630647 15243 leader_election.cc:302] T 5368e56e23d842d290c27a3aaf5cc7ef P 2be19306ea2f4530b6865458260943c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2be19306ea2f4530b6865458260943c1, 445b33f7957e4331b6dd0d75717210c7; no voters:
I20260928 12:23:46.630751 15529 raft_consensus.cc:2781] T 5368e56e23d842d290c27a3aaf5cc7ef P 2be19306ea2f4530b6865458260943c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:23:46.630896 15529 raft_consensus.cc:686] T 5368e56e23d842d290c27a3aaf5cc7ef P 2be19306ea2f4530b6865458260943c1 [term 1 LEADER]: Becoming Leader. State: Replica: 2be19306ea2f4530b6865458260943c1, State: Running, Role: LEADER
I20260928 12:23:46.631007 15529 consensus_queue.cc:237] T 5368e56e23d842d290c27a3aaf5cc7ef P 2be19306ea2f4530b6865458260943c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445b33f7957e4331b6dd0d75717210c7" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 32865 } } peers { permanent_uuid: "7ab389dd4f6f49ec9ce84efe9758567a" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44787 } } peers { permanent_uuid: "2be19306ea2f4530b6865458260943c1" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 42915 } }
I20260928 12:23:46.631913 15021 catalog_manager.cc:5495] T 5368e56e23d842d290c27a3aaf5cc7ef P 2be19306ea2f4530b6865458260943c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2be19306ea2f4530b6865458260943c1 (127.2.56.2). New cstate: current_term: 1 leader_uuid: "2be19306ea2f4530b6865458260943c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445b33f7957e4331b6dd0d75717210c7" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 32865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ab389dd4f6f49ec9ce84efe9758567a" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be19306ea2f4530b6865458260943c1" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 42915 } health_report { overall_health: HEALTHY } } }
I20260928 12:23:46.653023 15520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:46.650326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55626 (local address 127.2.56.2:42915)
0928 12:23:46.650373 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:46.650375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:46.650384 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:23:46.650424 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:46.650425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:46.650447 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:46.650513 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:23:46.650712 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:46.651224 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:46.651970 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:46.652150 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:46.652203 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:46.652609 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:23:46.652726 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:23:46.652808 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:46.652849 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 12:23:46.652943 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 12:23:46.655180 15172 raft_consensus.cc:1261] T 5368e56e23d842d290c27a3aaf5cc7ef P 7ab389dd4f6f49ec9ce84efe9758567a [term 1 FOLLOWER]: Refusing update from remote peer 2be19306ea2f4530b6865458260943c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:23:46.655180 15426 raft_consensus.cc:1261] T 5368e56e23d842d290c27a3aaf5cc7ef P 445b33f7957e4331b6dd0d75717210c7 [term 1 FOLLOWER]: Refusing update from remote peer 2be19306ea2f4530b6865458260943c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:23:46.655515 15529 consensus_queue.cc:1035] T 5368e56e23d842d290c27a3aaf5cc7ef P 2be19306ea2f4530b6865458260943c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "445b33f7957e4331b6dd0d75717210c7" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 32865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:23:46.655664 15534 consensus_queue.cc:1035] T 5368e56e23d842d290c27a3aaf5cc7ef P 2be19306ea2f4530b6865458260943c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab389dd4f6f49ec9ce84efe9758567a" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:23:46.658624 15539 mvcc.cc:205] Tried to move back new op lower bound from 7334290336377167872 to 7334290336281296896. Current Snapshot: MvccSnapshot[applied={T|T < 7334290336377167872}]
I20260928 12:23:46.660120 15542 mvcc.cc:205] Tried to move back new op lower bound from 7334290336377167872 to 7334290336281296896. Current Snapshot: MvccSnapshot[applied={T|T < 7334290336377167872}]
I20260928 12:23:46.660739 15541 mvcc.cc:205] Tried to move back new op lower bound from 7334290336377167872 to 7334290336281296896. Current Snapshot: MvccSnapshot[applied={T|T < 7334290336377167872}]
I20260928 12:23:46.662411 15021 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:23:46.662506 15021 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:23:46.663355 15021 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 47feda07ac204242bb669b26dd56bdc6: Sending DeleteTablet for 3 replicas of tablet 5368e56e23d842d290c27a3aaf5cc7ef
I20260928 12:23:46.663743 15154 tablet_service.cc:1514] Processing DeleteTablet for tablet 5368e56e23d842d290c27a3aaf5cc7ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:23:46 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:59212
I20260928 12:23:46.663976 15288 tablet_service.cc:1514] Processing DeleteTablet for tablet 5368e56e23d842d290c27a3aaf5cc7ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:23:46 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:55622
I20260928 12:23:46.664031 15407 tablet_service.cc:1514] Processing DeleteTablet for tablet 5368e56e23d842d290c27a3aaf5cc7ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:23:46 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:35616
I20260928 12:23:46.664476 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 15060
I20260928 12:23:46.664510 15554 tablet_replica.cc:330] T 5368e56e23d842d290c27a3aaf5cc7ef P 2be19306ea2f4530b6865458260943c1: stopping tablet replica
I20260928 12:23:46.665139 15554 raft_consensus.cc:2229] T 5368e56e23d842d290c27a3aaf5cc7ef P 2be19306ea2f4530b6865458260943c1 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:23:46.665337 15554 raft_consensus.cc:2258] T 5368e56e23d842d290c27a3aaf5cc7ef P 2be19306ea2f4530b6865458260943c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:23:46.665903 15554 ts_tablet_manager.cc:1905] T 5368e56e23d842d290c27a3aaf5cc7ef P 2be19306ea2f4530b6865458260943c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:23:46.667685 15555 tablet_replica.cc:330] T 5368e56e23d842d290c27a3aaf5cc7ef P 445b33f7957e4331b6dd0d75717210c7: stopping tablet replica
I20260928 12:23:46.667840 15554 ts_tablet_manager.cc:1918] T 5368e56e23d842d290c27a3aaf5cc7ef P 2be19306ea2f4530b6865458260943c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:23:46.668102 15554 log.cc:1196] T 5368e56e23d842d290c27a3aaf5cc7ef P 2be19306ea2f4530b6865458260943c1: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-1/wal/wals/5368e56e23d842d290c27a3aaf5cc7ef
I20260928 12:23:46.668417 15554 ts_tablet_manager.cc:1939] T 5368e56e23d842d290c27a3aaf5cc7ef P 2be19306ea2f4530b6865458260943c1: Deleting consensus metadata
I20260928 12:23:46.668877 15009 catalog_manager.cc:4841] TS 2be19306ea2f4530b6865458260943c1 (127.2.56.2:42915): tablet 5368e56e23d842d290c27a3aaf5cc7ef (table test-table [id=50a558dba05a49acac8b66bf168b6ef6]) successfully deleted
I20260928 12:23:46.669142 15555 raft_consensus.cc:2229] T 5368e56e23d842d290c27a3aaf5cc7ef P 445b33f7957e4331b6dd0d75717210c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:23:46.669333 15555 raft_consensus.cc:2258] T 5368e56e23d842d290c27a3aaf5cc7ef P 445b33f7957e4331b6dd0d75717210c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:23:46.669870 15555 ts_tablet_manager.cc:1905] T 5368e56e23d842d290c27a3aaf5cc7ef P 445b33f7957e4331b6dd0d75717210c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:23:46.671453 15555 ts_tablet_manager.cc:1918] T 5368e56e23d842d290c27a3aaf5cc7ef P 445b33f7957e4331b6dd0d75717210c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:23:46.671569 15555 log.cc:1196] T 5368e56e23d842d290c27a3aaf5cc7ef P 445b33f7957e4331b6dd0d75717210c7: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598072638673-2272-0/minicluster-data/ts-2/wal/wals/5368e56e23d842d290c27a3aaf5cc7ef
I20260928 12:23:46.671872 15555 ts_tablet_manager.cc:1939] T 5368e56e23d842d290c27a3aaf5cc7ef P 445b33f7957e4331b6dd0d75717210c7: Deleting consensus metadata
W20260928 12:23:46.672045 15007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.56.1:44787 (error 108)
I20260928 12:23:46.672676 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 15224
I20260928 12:23:46.672694 15008 catalog_manager.cc:4841] TS 445b33f7957e4331b6dd0d75717210c7 (127.2.56.3:32865): tablet 5368e56e23d842d290c27a3aaf5cc7ef (table test-table [id=50a558dba05a49acac8b66bf168b6ef6]) successfully deleted
I20260928 12:23:46.675012 15515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:46.674872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:44787 (local address 127.0.0.1:59214)
0928 12:23:46.674934 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:46.674977 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.1:44787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 12:23:46.675168 15007 catalog_manager.cc:4584] TS 7ab389dd4f6f49ec9ce84efe9758567a (127.2.56.1:44787): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5368e56e23d842d290c27a3aaf5cc7ef: Network error: Client connection negotiation failed: client connection to 127.2.56.1:44787: connect: Connection refused (error 111)
I20260928 12:23:46.677124 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 15357
I20260928 12:23:46.680895 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 14989
I20260928 12:23:46.687152 2272 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 12:23:47.021554 2272 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 12:23:47.021654 2272 mini_ranger.cc:66] Stopping Ranger...
I20260928 12:23:47.356544 2272 mini_ranger.cc:68] Stopped Ranger
2026-09-28 12:23:47.356 UTC [14056] LOG: received smart shutdown request
2026-09-28 12:23:47.360 UTC [14056] LOG: background worker "logical replication launcher" (PID 14065) exited with exit code 1
2026-09-28 12:23:47.361 UTC [14060] LOG: shutting down
2026-09-28 12:23:47.539 UTC [14056] LOG: database system is shut down
2026-09-28T12:23:47Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47559 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:23:47 dist-test-slave-920p krb5kdc[15565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:23:47 dist-test-slave-920p krb5kdc[15565](info): set up 2 sockets
Sep 28 12:23:47 dist-test-slave-920p krb5kdc[15565](info): commencing operation
krb5kdc: starting...
W20260928 12:23:48.597167 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:23:48 dist-test-slave-920p krb5kdc[15565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:23:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:23:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
I20260928 12:23:48.714463 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:34799
--webserver_interface=127.2.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:37099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:34799,127.2.56.61:33199,127.2.56.60:46431
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:23:48.793994 15578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:23:48.794196 15578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:23:48.794225 15578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:23:48.795348 15578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:23:48.795387 15578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:23:48.795404 15578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:23:48.795420 15578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:23:48.795435 15578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:23:48.796937 15578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:37099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:34799,127.2.56.61:33199,127.2.56.60:46431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:34799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:23:48.797176 15578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:23:48.797399 15578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:23:48.801931 15584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:23:48.801945 15586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:23:48.802048 15578 server_base.cc:1034] running on GCE node
W20260928 12:23:48.801968 15583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:23:48.802386 15578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:23:48.802661 15578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:23:48.803802 15578 hybrid_clock.cc:648] HybridClock initialized: now 1790598228803789 us; error 30 us; skew 500 ppm
Sep 28 12:23:48 dist-test-slave-920p krb5kdc[15565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598228, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:23:48.806231 15578 init.cc:377] Logged in from keytab as kudu/127.2.56.62@KRBTEST.COM (short username kudu)
I20260928 12:23:48.806653 15578 webserver.cc:460] Webserver started at http://127.2.56.62:42645/ using document root <none> and password file <none>
I20260928 12:23:48.806842 15578 fs_manager.cc:359] Metadata directory not provided
I20260928 12:23:48.806890 15578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:23:48.807011 15578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:23:48.808010 15578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "223b565174054d7fa706d88ce7edcdf6"
format_stamp: "Formatted at 2026-09-28 12:23:48 on dist-test-slave-920p"
server_key: "07f1746f6865937dcf20c475579bfcc3"
server_key_iv: "7162253b075b894475ccf089e2a390cc"
server_key_version: "encryptionkey@0"
I20260928 12:23:48.808327 15578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "223b565174054d7fa706d88ce7edcdf6"
format_stamp: "Formatted at 2026-09-28 12:23:48 on dist-test-slave-920p"
server_key: "07f1746f6865937dcf20c475579bfcc3"
server_key_iv: "7162253b075b894475ccf089e2a390cc"
server_key_version: "encryptionkey@0"
I20260928 12:23:48.809645 15578 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 12:23:48.810309 15593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:48.810447 15578 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:48.810532 15578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "223b565174054d7fa706d88ce7edcdf6"
format_stamp: "Formatted at 2026-09-28 12:23:48 on dist-test-slave-920p"
server_key: "07f1746f6865937dcf20c475579bfcc3"
server_key_iv: "7162253b075b894475ccf089e2a390cc"
server_key_version: "encryptionkey@0"
I20260928 12:23:48.810596 15578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:23:48.819131 15578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:23:48.820766 15578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:23:48.820891 15578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:23:48.824553 15578 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:34799
I20260928 12:23:48.824586 15645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:34799 every 8 connection(s)
I20260928 12:23:48.824890 15578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:23:48.825742 15646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:23:48.827211 15646 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34799 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:23:48.829085 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 15578
I20260928 12:23:48.829169 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:23:48.829382 2272 external_mini_cluster.cc:1428] Setting key 2ddb5e45424fb957e50aee5f7db1d6e9
Sep 28 12:23:48 dist-test-slave-920p krb5kdc[15565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598228, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.56.61@KRBTEST.COM; defaulting to no policy
I20260928 12:23:48.837455 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:48.827903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54610 (local address 127.2.56.62:34799)
0928 12:23:48.828040 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:48.828048 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:48.828201 (+ 153us) server_negotiation.cc:407] Connection header received
0928 12:23:48.828685 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:48.828687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:48.828711 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:48.828787 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:23:48.829328 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:48.830131 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:48.831330 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:48.831640 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:48.831729 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:48.834070 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:23:48.834080 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:23:48.834080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:23:48.834088 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:23:48.835850 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:48.836201 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:48.836204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:48.836205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:48.836305 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:48.836521 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:48.836522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:48.836522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:48.836812 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:23:48.836896 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:48.837069 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 12:23:48.837237 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":104,"thread_start_us":65,"threads_started":1}
I20260928 12:23:48.837458 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:48.827867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:34799 (local address 127.0.0.1:54610)
0928 12:23:48.828098 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:48.828125 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:48.828564 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:48.828805 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:48.828807 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:48.829043 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:23:48.829222 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:48.829227 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:48.830215 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:48.830216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:48.831234 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:48.831238 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:48.831715 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:48.831716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:48.831773 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:48.832307 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:23:48.832315 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:23:48.833985 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:23:48.835934 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:48.835941 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:48.835942 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:48.836123 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:48.836385 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:48.836386 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:48.836386 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:48.836461 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:48.836879 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:23:48.836884 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:23:48.836949 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 12:23:48.837019 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 12:23:48.837178 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":91,"threads_started":1}
I20260928 12:23:48.838150 15646 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.56.61" port: 33199 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:23:48.838511 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:48.838409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.61:33199 (local address 127.0.0.1:49434)
0928 12:23:48.838451 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:48.838492 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.61:33199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 12:23:48.838632 15597 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.56.61:33199: connect: Connection refused (error 111)
I20260928 12:23:48.839051 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:48.838975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.61:33199 (local address 127.0.0.1:49450)
0928 12:23:48.839012 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:48.839035 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.61:33199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 12:23:48.839149 15646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.56.61:33199: Network error: Client connection negotiation failed: client connection to 127.2.56.61:33199: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.61.keytab.
Entry for principal kudu/127.2.56.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.61.keytab.
I20260928 12:23:48.854162 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.61:33199
--webserver_interface=127.2.56.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.61.keytab
--principal=kudu/127.2.56.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:37099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:34799,127.2.56.61:33199,127.2.56.60:46431
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:23:48.888389 15646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.56.61" port: 33199 } attempt: 1
I20260928 12:23:48.888953 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:48.888766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.61:33199 (local address 127.0.0.1:49456)
0928 12:23:48.888850 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:48.888909 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.61:33199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 12:23:48.889341 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:48.889205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.61:33199 (local address 127.0.0.1:49472)
0928 12:23:48.889252 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:48.889314 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.61:33199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 12:23:48.889442 15646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.56.61:33199: Network error: Client connection negotiation failed: client connection to 127.2.56.61:33199: connect: Connection refused (error 111)
W20260928 12:23:48.933169 15653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:23:48.933387 15653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:23:48.933409 15653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:23:48.934525 15653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:23:48.934563 15653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:23:48.934576 15653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:23:48.934587 15653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:23:48.934597 15653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:23:48.936061 15653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:37099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:34799,127.2.56.61:33199,127.2.56.60:46431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.61.keytab
--principal=kudu/127.2.56.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.61:33199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.56.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:23:48.936273 15653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:23:48.936475 15653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:23:48.940938 15661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:23:48.940953 15658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:23:48.941118 15653 server_base.cc:1034] running on GCE node
W20260928 12:23:48.940953 15659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:23:48.941442 15653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:23:48.941717 15653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:23:48.942862 15653 hybrid_clock.cc:648] HybridClock initialized: now 1790598228942848 us; error 28 us; skew 500 ppm
Sep 28 12:23:48 dist-test-slave-920p krb5kdc[15565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598228, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:23:48.945547 15653 init.cc:377] Logged in from keytab as kudu/127.2.56.61@KRBTEST.COM (short username kudu)
I20260928 12:23:48.945921 15653 webserver.cc:460] Webserver started at http://127.2.56.61:40657/ using document root <none> and password file <none>
I20260928 12:23:48.946095 15653 fs_manager.cc:359] Metadata directory not provided
I20260928 12:23:48.946137 15653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:23:48.946226 15653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:23:48.947219 15653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-1/data/instance:
uuid: "7cfc9d0cbd72480995e129eac1dc993a"
format_stamp: "Formatted at 2026-09-28 12:23:48 on dist-test-slave-920p"
server_key: "ea82d60d1c349e6dfcfb5a07e56d9e86"
server_key_iv: "2a045c284f91f69d968c47e46b4fe316"
server_key_version: "encryptionkey@0"
I20260928 12:23:48.947510 15653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-1/wal/instance:
uuid: "7cfc9d0cbd72480995e129eac1dc993a"
format_stamp: "Formatted at 2026-09-28 12:23:48 on dist-test-slave-920p"
server_key: "ea82d60d1c349e6dfcfb5a07e56d9e86"
server_key_iv: "2a045c284f91f69d968c47e46b4fe316"
server_key_version: "encryptionkey@0"
I20260928 12:23:48.948725 15653 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 12:23:48.949365 15668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:48.949518 15653 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:48.949599 15653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-1/wal
uuid: "7cfc9d0cbd72480995e129eac1dc993a"
format_stamp: "Formatted at 2026-09-28 12:23:48 on dist-test-slave-920p"
server_key: "ea82d60d1c349e6dfcfb5a07e56d9e86"
server_key_iv: "2a045c284f91f69d968c47e46b4fe316"
server_key_version: "encryptionkey@0"
I20260928 12:23:48.949657 15653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:23:48.956137 15653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:23:48.957571 15646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.56.61" port: 33199 } attempt: 2
I20260928 12:23:48.957954 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:48.957833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.61:33199 (local address 127.0.0.1:49482)
0928 12:23:48.957889 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:48.957924 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.61:33199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 12:23:48.958110 15653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:23:48.958276 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:48.958182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.61:33199 (local address 127.0.0.1:49496)
0928 12:23:48.958233 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:48.958257 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.61:33199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 12:23:48.958262 15653 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260928 12:23:48.958367 15646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.56.61:33199: Network error: Client connection negotiation failed: client connection to 127.2.56.61:33199: connect: Connection refused (error 111)
I20260928 12:23:48.961915 15653 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.61:33199
I20260928 12:23:48.961941 15720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.61:33199 every 8 connection(s)
I20260928 12:23:48.962236 15653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-1/data/info.pb
I20260928 12:23:48.963027 15721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:23:48.964494 15721 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34799 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:23:48.968528 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 15653
I20260928 12:23:48.968640 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-1/wal/instance
I20260928 12:23:48.968823 2272 external_mini_cluster.cc:1428] Setting key c0a8fc27361eb447d6d1702dcf47b4ac
Sep 28 12:23:48 dist-test-slave-920p krb5kdc[15565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598228, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.61@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:23:48.973606 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:48.965146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54620 (local address 127.2.56.62:34799)
0928 12:23:48.965194 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:48.965196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:48.965406 (+ 210us) server_negotiation.cc:407] Connection header received
0928 12:23:48.965859 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:48.965859 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:48.965871 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:48.965919 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:23:48.966555 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:48.967036 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:48.968000 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:48.968311 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:48.968383 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:48.970861 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:23:48.970863 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:23:48.970863 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:23:48.970871 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:23:48.972278 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:48.972656 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:48.972657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:48.972658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:48.972714 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:48.972936 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:48.972937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:48.972937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:48.973057 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:23:48.973110 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:48.973425 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 12:23:48.973513 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 12:23:48.973770 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:48.965118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:34799 (local address 127.0.0.1:54620)
0928 12:23:48.965285 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:48.965330 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:48.965761 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:48.965972 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:48.965975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:48.966216 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:23:48.966460 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:48.966466 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:48.967112 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:48.967113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:48.967912 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:48.967914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:48.968430 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:48.968431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:48.968590 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:48.969115 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:23:48.969124 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:23:48.970746 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:23:48.972356 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:48.972364 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:48.972366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:48.972591 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:48.972766 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:48.972766 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:48.972767 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:48.972881 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:48.973110 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:23:48.973116 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:23:48.973284 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 12:23:48.973395 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 12:23:48.973550 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260928 12:23:48.974290 15721 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.56.61" port: 33199 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.56.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.60@KRBTEST.COM" created.
Sep 28 12:23:48 dist-test-slave-920p krb5kdc[15565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598228, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.61@KRBTEST.COM for kudu/127.2.56.61@KRBTEST.COM
I20260928 12:23:48.981886 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:48.974529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.61:33199 (local address 127.0.0.1:49506)
0928 12:23:48.974581 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:48.974593 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:48.974642 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:48.974869 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:48.974870 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:48.975015 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:23:48.975081 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:48.975082 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:48.975960 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:48.975961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:48.976461 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:48.976462 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:48.976964 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:48.976965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:48.977031 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:48.977410 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:23:48.977414 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:23:48.979102 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:23:48.980881 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:48.980881 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:48.980882 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:48.981037 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:48.981208 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:48.981209 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:48.981209 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:48.981260 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:48.981575 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:23:48.981576 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:23:48.981612 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 12:23:48.981665 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 12:23:48.981769 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 12:23:48.981946 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:48.974576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49506 (local address 127.2.56.61:33199)
0928 12:23:48.974737 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:48.974742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:48.974759 (+ 17us) server_negotiation.cc:407] Connection header received
0928 12:23:48.974779 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:48.974780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:48.974791 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:48.974848 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:23:48.975146 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:48.975892 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:48.976520 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:48.976884 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:48.976938 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:48.979188 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:23:48.979199 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:23:48.979199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:23:48.979206 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:23:48.980791 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:48.981106 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:48.981110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:48.981110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:48.981157 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:48.981327 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:48.981328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:48.981329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:48.981476 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:23:48.981554 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:48.981710 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 12:23:48.981819 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":133,"thread_start_us":98,"threads_started":1}
I20260928 12:23:48.982550 15721 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.56.60" port: 46431 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:23:48.982900 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:48.982794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:46431 (local address 127.0.0.1:42080)
0928 12:23:48.982833 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:48.982876 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.60:46431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 12:23:48.983009 15670 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.56.60:46431: connect: Connection refused (error 111)
I20260928 12:23:48.983492 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:48.983390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:46431 (local address 127.0.0.1:42086)
0928 12:23:48.983429 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:48.983468 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.60:46431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 12:23:48.983605 15721 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.56.60:46431: Network error: Client connection negotiation failed: client connection to 127.2.56.60:46431: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.60.keytab.
Entry for principal kudu/127.2.56.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.60.keytab.
I20260928 12:23:48.992274 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.60:46431
--webserver_interface=127.2.56.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.60.keytab
--principal=kudu/127.2.56.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:37099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:34799,127.2.56.61:33199,127.2.56.60:46431
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:23:49.032869 15721 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.56.60" port: 46431 } attempt: 1
I20260928 12:23:49.033427 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:49.033249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:46431 (local address 127.0.0.1:42100)
0928 12:23:49.033345 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:49.033391 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.60:46431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 12:23:49.033751 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:49.033664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:46431 (local address 127.0.0.1:42110)
0928 12:23:49.033705 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:49.033732 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.60:46431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 12:23:49.033839 15721 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.56.60:46431: Network error: Client connection negotiation failed: client connection to 127.2.56.60:46431: connect: Connection refused (error 111)
I20260928 12:23:49.049520 15646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.56.61" port: 33199 } attempt: 3
Sep 28 12:23:49 dist-test-slave-920p krb5kdc[15565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598228, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for kudu/127.2.56.61@KRBTEST.COM
I20260928 12:23:49.056402 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:49.049859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.61:33199 (local address 127.0.0.1:49518)
0928 12:23:49.049925 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:49.049940 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:49.050005 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:49.050140 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:49.050140 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:49.050322 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:23:49.050407 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:49.050410 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:49.051091 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:49.051092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:49.051902 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:49.051907 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:49.052329 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:49.052330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:49.052409 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:49.052857 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:23:49.052862 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:23:49.054137 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:23:49.055529 (+ 1392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:49.055530 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:49.055530 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:49.055684 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:49.055846 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:49.055847 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:49.055847 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:49.055902 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:49.056100 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:23:49.056101 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:23:49.056146 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 12:23:49.056204 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 12:23:49.056298 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 12:23:49.056402 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:49.049937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49518 (local address 127.2.56.61:33199)
0928 12:23:49.049991 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:49.049993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:49.050004 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:23:49.050068 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:49.050068 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:49.050077 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:49.050123 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:23:49.050504 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:49.051014 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:49.051983 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:49.052261 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:49.052313 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:49.054214 (+ 1901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:23:49.054215 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:23:49.054216 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:23:49.054222 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:23:49.055457 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:49.055741 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:49.055742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:49.055743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:49.055787 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:49.055949 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:49.055950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:49.055950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:49.056049 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:23:49.056101 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:49.056231 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 12:23:49.056299 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 12:23:49.056912 15646 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.56.60" port: 46431 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:23:49.057241 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:49.057143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:46431 (local address 127.0.0.1:42114)
0928 12:23:49.057185 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:49.057218 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.60:46431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 12:23:49.057524 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:49.057464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:46431 (local address 127.0.0.1:42122)
0928 12:23:49.057490 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:49.057511 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.60:46431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260928 12:23:49.057607 15646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.56.60:46431: Network error: Client connection negotiation failed: client connection to 127.2.56.60:46431: connect: Connection refused (error 111)
W20260928 12:23:49.075706 15728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:23:49.075932 15728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:23:49.075964 15728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:23:49.077128 15728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:23:49.077168 15728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:23:49.077186 15728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:23:49.077203 15728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:23:49.077219 15728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:23:49.078825 15728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:37099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:34799,127.2.56.61:33199,127.2.56.60:46431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.60.keytab
--principal=kudu/127.2.56.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.60:46431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.56.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:23:49.079077 15728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:23:49.079394 15728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:23:49.084739 15728 server_base.cc:1034] running on GCE node
W20260928 12:23:49.084722 15733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:23:49.084731 15734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:23:49.084722 15736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:23:49.085155 15728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:23:49.085498 15728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:23:49.086669 15728 hybrid_clock.cc:648] HybridClock initialized: now 1790598229086649 us; error 35 us; skew 500 ppm
Sep 28 12:23:49 dist-test-slave-920p krb5kdc[15565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598229, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:23:49.088748 15646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.56.60" port: 46431 } attempt: 1
I20260928 12:23:49.089097 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:49.088984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:46431 (local address 127.0.0.1:42130)
0928 12:23:49.089040 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:49.089072 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.60:46431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 12:23:49.089222 15728 init.cc:377] Logged in from keytab as kudu/127.2.56.60@KRBTEST.COM (short username kudu)
I20260928 12:23:49.089493 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:49.089387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:46431 (local address 127.0.0.1:42146)
0928 12:23:49.089433 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:49.089465 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.60:46431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 12:23:49.089577 15646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.56.60:46431: Network error: Client connection negotiation failed: client connection to 127.2.56.60:46431: connect: Connection refused (error 111)
I20260928 12:23:49.089769 15728 webserver.cc:460] Webserver started at http://127.2.56.60:42267/ using document root <none> and password file <none>
I20260928 12:23:49.089975 15728 fs_manager.cc:359] Metadata directory not provided
I20260928 12:23:49.090020 15728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:23:49.090124 15728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:23:49.091168 15728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-2/data/instance:
uuid: "82e827cf3dac4a34930c1e2e363897d6"
format_stamp: "Formatted at 2026-09-28 12:23:49 on dist-test-slave-920p"
server_key: "3538de359ee15152b045b64338820677"
server_key_iv: "cb36e45c97a138f74235e4737e723655"
server_key_version: "encryptionkey@0"
I20260928 12:23:49.091512 15728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-2/wal/instance:
uuid: "82e827cf3dac4a34930c1e2e363897d6"
format_stamp: "Formatted at 2026-09-28 12:23:49 on dist-test-slave-920p"
server_key: "3538de359ee15152b045b64338820677"
server_key_iv: "cb36e45c97a138f74235e4737e723655"
server_key_version: "encryptionkey@0"
I20260928 12:23:49.092694 15728 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:23:49.093449 15743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:49.093585 15728 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:23:49.093664 15728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-2/wal
uuid: "82e827cf3dac4a34930c1e2e363897d6"
format_stamp: "Formatted at 2026-09-28 12:23:49 on dist-test-slave-920p"
server_key: "3538de359ee15152b045b64338820677"
server_key_iv: "cb36e45c97a138f74235e4737e723655"
server_key_version: "encryptionkey@0"
I20260928 12:23:49.093722 15728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:23:49.101969 15721 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.56.60" port: 46431 } attempt: 2
I20260928 12:23:49.102437 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:49.102289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:46431 (local address 127.0.0.1:42156)
0928 12:23:49.102352 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:49.102397 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.60:46431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 12:23:49.102857 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:49.102736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:46431 (local address 127.0.0.1:42166)
0928 12:23:49.102798 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:49.102832 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.60:46431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 12:23:49.102955 15721 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.56.60:46431: Network error: Client connection negotiation failed: client connection to 127.2.56.60:46431: connect: Connection refused (error 111)
I20260928 12:23:49.108664 15728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:23:49.110807 15728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:23:49.110939 15728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:23:49.114739 15728 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.60:46431
I20260928 12:23:49.114781 15795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.60:46431 every 8 connection(s)
I20260928 12:23:49.115088 15728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-2/data/info.pb
I20260928 12:23:49.115898 15796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:23:49.117664 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 15728
I20260928 12:23:49.117524 15796 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34799 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:23:49.117753 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598072638673-2272-0/minicluster-data/master-2/wal/instance
I20260928 12:23:49.117904 2272 external_mini_cluster.cc:1428] Setting key 1f12f41fb4cb7b789a6f9c6912a82c5d
Sep 28 12:23:49 dist-test-slave-920p krb5kdc[15565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598229, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.60@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
Sep 28 12:23:49 dist-test-slave-920p krb5kdc[15565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.60@KRBTEST.COM
I20260928 12:23:49.126106 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:49.118297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54628 (local address 127.2.56.62:34799)
0928 12:23:49.118371 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:49.118373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:49.118591 (+ 218us) server_negotiation.cc:407] Connection header received
0928 12:23:49.119020 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:49.119021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:49.119032 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:49.119079 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:23:49.119625 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:49.120088 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:49.120941 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:49.121125 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:49.121180 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:49.123332 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:23:49.123333 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:23:49.123333 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:23:49.123341 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:23:49.124706 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:49.125106 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:49.125108 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:49.125108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:49.125165 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:49.125454 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:49.125455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:49.125455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:49.125575 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:23:49.125638 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:49.125929 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 12:23:49.126010 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 12:23:49.126261 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:49.118240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:34799 (local address 127.0.0.1:54628)
0928 12:23:49.118494 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:49.118531 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:49.118943 (+ 412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:49.119106 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:49.119109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:49.119381 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:23:49.119540 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:49.119544 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:49.120153 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:49.120154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:49.120863 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:49.120865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:49.121190 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:49.121192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:49.121350 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:49.121808 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:23:49.121818 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:23:49.123250 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:23:49.124782 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:49.124788 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:49.124790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:49.125038 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:49.125223 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:49.125223 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:49.125223 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:49.125395 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:49.125646 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:23:49.125651 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:23:49.125802 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 12:23:49.125889 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 12:23:49.126027 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":1,"thread_start_us":76,"threads_started":1}
I20260928 12:23:49.126377 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:49.118338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42172 (local address 127.2.56.60:46431)
0928 12:23:49.118479 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:49.118488 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:49.118514 (+ 26us) server_negotiation.cc:407] Connection header received
0928 12:23:49.118930 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:49.118936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:49.118956 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:49.119049 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:23:49.119604 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:49.120364 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:49.121061 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:49.121427 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:49.121484 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:49.123689 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:23:49.123695 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:23:49.123696 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:23:49.123702 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:23:49.125198 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:49.125543 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:49.125545 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:49.125546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:49.125597 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:49.125759 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:49.125760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:49.125760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:49.125890 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:23:49.126019 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:49.126090 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 12:23:49.126156 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":106,"thread_start_us":77,"threads_started":1}
I20260928 12:23:49.126926 15796 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.56.61" port: 33199 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 12:23:49 dist-test-slave-920p krb5kdc[15565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598229, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.60@KRBTEST.COM for kudu/127.2.56.61@KRBTEST.COM
Sep 28 12:23:49 dist-test-slave-920p krb5kdc[15565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:23:49.133358 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:49.127164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.61:33199 (local address 127.0.0.1:49524)
0928 12:23:49.127202 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:49.127210 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:49.127237 (+ 27us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:49.127406 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:49.127407 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:49.127508 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:23:49.127542 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:49.127544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:49.128217 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:49.128218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:49.128798 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:49.128800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:49.129112 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:49.129113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:49.129166 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:49.129551 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:23:49.129556 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:23:49.130723 (+ 1167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:23:49.132407 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:49.132408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:49.132408 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:49.132594 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:49.132780 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:49.132781 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:49.132781 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:49.132829 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:49.133064 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:23:49.133065 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:23:49.133098 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 12:23:49.133156 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 12:23:49.133240 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 12:23:49.133394 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:49.127226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49524 (local address 127.2.56.61:33199)
0928 12:23:49.127294 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:49.127296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:49.127304 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:23:49.127328 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:49.127330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:49.127341 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:49.127406 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:23:49.127606 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:49.128149 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:49.128884 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:49.129062 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:49.129105 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:49.130801 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:23:49.130802 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:23:49.130802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:23:49.130808 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:23:49.132316 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:49.132660 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:49.132661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:49.132661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:49.132715 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:49.132879 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:49.132880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:49.132880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:49.133012 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:23:49.133075 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:49.133207 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 12:23:49.133307 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 12:23:49.133888 15796 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.56.60" port: 46431 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:23:49.134277 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:49.127416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54632 (local address 127.2.56.62:34799)
0928 12:23:49.127454 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:49.127456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:49.127462 (+ 6us) server_negotiation.cc:407] Connection header received
0928 12:23:49.127478 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:49.127478 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:49.127485 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:49.127525 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:23:49.127964 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:49.128486 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:49.129228 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:49.129464 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:49.129517 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:49.131633 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:23:49.131634 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:23:49.131634 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:23:49.131641 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:23:49.133073 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:49.133396 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:49.133397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:49.133398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:49.133450 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:49.133626 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:49.133627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:49.133627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:49.133749 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:23:49.133891 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:49.134113 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 12:23:49.134195 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 12:23:49 dist-test-slave-920p krb5kdc[15565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598229, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.60@KRBTEST.COM for kudu/127.2.56.60@KRBTEST.COM
Sep 28 12:23:49 dist-test-slave-920p krb5kdc[15565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.61@KRBTEST.COM
I20260928 12:23:49.139801 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:49.134129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:46431 (local address 127.0.0.1:42182)
0928 12:23:49.134224 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:49.134233 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:49.134267 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:49.134381 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:49.134382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:49.134523 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:23:49.134553 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:49.134555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:49.135266 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:49.135267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:49.135734 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:49.135736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:49.136051 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:49.136051 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:49.136094 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:49.136433 (+ 339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:23:49.136441 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:23:49.137571 (+ 1130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:23:49.138865 (+ 1294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:49.138866 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:49.138866 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:49.139018 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:49.139190 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:49.139190 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:49.139191 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:49.139243 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:49.139476 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:23:49.139476 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:23:49.139513 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 12:23:49.139578 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 12:23:49.139666 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 12:23:49.139843 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:49.134183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42182 (local address 127.2.56.60:46431)
0928 12:23:49.134235 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:49.134237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:49.134254 (+ 17us) server_negotiation.cc:407] Connection header received
0928 12:23:49.134309 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:49.134310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:49.134320 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:49.134370 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:23:49.134619 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:49.135201 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:49.135796 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:49.135998 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:49.136049 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:49.137640 (+ 1591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:23:49.137641 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:23:49.137642 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:23:49.137647 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:23:49.138790 (+ 1143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:49.139082 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:49.139083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:49.139084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:49.139136 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:49.139307 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:49.139308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:49.139308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:49.139419 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:23:49.139478 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:49.139597 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 12:23:49.139710 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":34}
I20260928 12:23:49.141486 15796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82e827cf3dac4a34930c1e2e363897d6: Bootstrap starting.
I20260928 12:23:49.141767 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:49.134941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49538 (local address 127.2.56.61:33199)
0928 12:23:49.134978 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:49.134979 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:49.135020 (+ 41us) server_negotiation.cc:407] Connection header received
0928 12:23:49.135075 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:49.135075 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:49.135084 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:49.135133 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:23:49.135538 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:49.135979 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:49.136700 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:49.136970 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:49.137029 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:49.139101 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:23:49.139103 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:23:49.139103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:23:49.139109 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:23:49.140646 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:49.140929 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:49.140930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:49.140930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:49.141004 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:49.141173 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:49.141174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:49.141174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:49.141324 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:23:49.141500 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:49.141539 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 12:23:49.141691 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 12:23:49.142428 15796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82e827cf3dac4a34930c1e2e363897d6: Neither blocks nor log segments found. Creating new log.
I20260928 12:23:49.142691 15796 log.cc:824] T 00000000000000000000000000000000 P 82e827cf3dac4a34930c1e2e363897d6: Log is configured to *not* fsync() on all Append() calls
I20260928 12:23:49.143105 2272 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:23:49.143440 15796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82e827cf3dac4a34930c1e2e363897d6: No bootstrap required, opened a new log
I20260928 12:23:49.145292 15796 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82e827cf3dac4a34930c1e2e363897d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223b565174054d7fa706d88ce7edcdf6" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34799 } } peers { permanent_uuid: "7cfc9d0cbd72480995e129eac1dc993a" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 33199 } } peers { permanent_uuid: "82e827cf3dac4a34930c1e2e363897d6" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 46431 } }
I20260928 12:23:49.145555 15796 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82e827cf3dac4a34930c1e2e363897d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:23:49.145644 15796 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82e827cf3dac4a34930c1e2e363897d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82e827cf3dac4a34930c1e2e363897d6, State: Initialized, Role: FOLLOWER
I20260928 12:23:49.145823 15796 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82e827cf3dac4a34930c1e2e363897d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223b565174054d7fa706d88ce7edcdf6" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34799 } } peers { permanent_uuid: "7cfc9d0cbd72480995e129eac1dc993a" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 33199 } } peers { permanent_uuid: "82e827cf3dac4a34930c1e2e363897d6" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 46431 } }
I20260928 12:23:49.146234 15796 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82e827cf3dac4a34930c1e2e363897d6 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 12:23:49.148092 15820 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 82e827cf3dac4a34930c1e2e363897d6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:23:49.148185 15820 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:23:49.148437 15809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82e827cf3dac4a34930c1e2e363897d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223b565174054d7fa706d88ce7edcdf6" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34799 } } peers { permanent_uuid: "7cfc9d0cbd72480995e129eac1dc993a" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 33199 } } peers { permanent_uuid: "82e827cf3dac4a34930c1e2e363897d6" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 46431 } } }
I20260928 12:23:49.148592 15809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82e827cf3dac4a34930c1e2e363897d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:23:49.151340 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:49.144279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54634 (local address 127.2.56.62:34799)
0928 12:23:49.144377 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:49.144379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:49.144489 (+ 110us) server_negotiation.cc:407] Connection header received
0928 12:23:49.144578 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:49.144579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:49.144589 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:49.144615 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:23:49.145100 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:49.145733 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:49.146701 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:49.146984 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:49.147034 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:49.148393 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:23:49.148395 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:23:49.148395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:23:49.148402 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:23:49.150064 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:49.150467 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:49.150468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:49.150469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:49.150524 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:49.150695 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:49.150696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:49.150696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:49.150817 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:23:49.150908 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:49.151187 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 12:23:49.151255 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 12:23:49.148612 15728 master_runner.cc:186] Error getting master registration for 127.2.56.62:34799: OK, instance_id { permanent_uuid: "223b565174054d7fa706d88ce7edcdf6" instance_seqno: 1790598228823543 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 12:23:49.151490 15728 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 12:23:49.152256 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:49.144317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49542 (local address 127.2.56.61:33199)
0928 12:23:49.144349 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:49.144351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:49.144662 (+ 311us) server_negotiation.cc:407] Connection header received
0928 12:23:49.144764 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:49.144765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:49.144773 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:49.144798 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:23:49.145459 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:49.146082 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:49.147045 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:49.147315 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:49.147356 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:49.149240 (+ 1884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:23:49.149241 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:23:49.149241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:23:49.149247 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:23:49.151167 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:49.151511 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:49.151512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:49.151513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:49.151563 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:49.151754 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:49.151756 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:49.151756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:49.151902 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:23:49.152005 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:49.152104 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 12:23:49.152172 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 12:23:49.153312 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:49.144447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42186 (local address 127.2.56.60:46431)
0928 12:23:49.144481 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:49.144483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:49.145406 (+ 923us) server_negotiation.cc:407] Connection header received
0928 12:23:49.145505 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:49.145505 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:49.145514 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:49.145538 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:23:49.146183 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:49.146855 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:49.148289 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:49.148552 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:49.148801 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:49.150722 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:23:49.150724 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:23:49.150724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:23:49.150731 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:23:49.152180 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:49.152535 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:49.152536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:49.152536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:49.152593 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:49.152803 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:49.152804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:49.152804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:49.152920 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:23:49.153015 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:49.153174 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 12:23:49.153230 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:23:49.164731 15646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.56.60" port: 46431 } attempt: 2
Sep 28 12:23:49 dist-test-slave-920p krb5kdc[15565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598228, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for kudu/127.2.56.60@KRBTEST.COM
I20260928 12:23:49.171669 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:49.165021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:46431 (local address 127.0.0.1:42188)
0928 12:23:49.165075 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:49.165088 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:49.165135 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:49.165305 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:49.165306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:49.165486 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:23:49.165535 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:49.165538 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:49.166174 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:49.166174 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:49.166997 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:49.167000 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:49.167367 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:49.167369 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:49.167454 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:49.167861 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:23:49.167866 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:23:49.169387 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:23:49.170777 (+ 1390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:49.170778 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:49.170779 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:49.170936 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:49.171096 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:49.171097 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:49.171097 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:49.171151 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:49.171365 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:23:49.171366 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:23:49.171412 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 12:23:49.171471 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 12:23:49.171564 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 12:23:49.171669 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:49.165077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42188 (local address 127.2.56.60:46431)
0928 12:23:49.165123 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:49.165124 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:49.165132 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:23:49.165195 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:49.165195 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:49.165203 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:49.165248 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:23:49.165629 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:49.166102 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:49.167075 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:49.167296 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:49.167348 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:49.169468 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:23:49.169470 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:23:49.169470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:23:49.169478 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:23:49.170701 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:49.170995 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:49.170996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:49.170997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:49.171043 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:49.171206 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:49.171207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:49.171208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:49.171309 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:23:49.171364 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:49.171500 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 12:23:49.171580 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 12:23:49.173017 15646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 223b565174054d7fa706d88ce7edcdf6: Bootstrap starting.
I20260928 12:23:49.173758 15646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 223b565174054d7fa706d88ce7edcdf6: Neither blocks nor log segments found. Creating new log.
I20260928 12:23:49.174013 15646 log.cc:824] T 00000000000000000000000000000000 P 223b565174054d7fa706d88ce7edcdf6: Log is configured to *not* fsync() on all Append() calls
I20260928 12:23:49.174674 15646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 223b565174054d7fa706d88ce7edcdf6: No bootstrap required, opened a new log
I20260928 12:23:49.176065 15646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 223b565174054d7fa706d88ce7edcdf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223b565174054d7fa706d88ce7edcdf6" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34799 } } peers { permanent_uuid: "7cfc9d0cbd72480995e129eac1dc993a" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 33199 } } peers { permanent_uuid: "82e827cf3dac4a34930c1e2e363897d6" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 46431 } }
I20260928 12:23:49.176177 15646 raft_consensus.cc:374] T 00000000000000000000000000000000 P 223b565174054d7fa706d88ce7edcdf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:23:49.176196 15646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 223b565174054d7fa706d88ce7edcdf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 223b565174054d7fa706d88ce7edcdf6, State: Initialized, Role: FOLLOWER
I20260928 12:23:49.176309 15646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 223b565174054d7fa706d88ce7edcdf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223b565174054d7fa706d88ce7edcdf6" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34799 } } peers { permanent_uuid: "7cfc9d0cbd72480995e129eac1dc993a" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 33199 } } peers { permanent_uuid: "82e827cf3dac4a34930c1e2e363897d6" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 46431 } }
I20260928 12:23:49.176548 15823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 223b565174054d7fa706d88ce7edcdf6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223b565174054d7fa706d88ce7edcdf6" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34799 } } peers { permanent_uuid: "7cfc9d0cbd72480995e129eac1dc993a" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 33199 } } peers { permanent_uuid: "82e827cf3dac4a34930c1e2e363897d6" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 46431 } } }
I20260928 12:23:49.176595 15646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 223b565174054d7fa706d88ce7edcdf6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:23:49.176635 15823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 223b565174054d7fa706d88ce7edcdf6 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 12:23:49.178246 15834 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 223b565174054d7fa706d88ce7edcdf6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:23:49.178351 15834 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:23:49.179139 15578 master_runner.cc:186] Error getting master registration for 127.2.56.61:33199: OK, instance_id { permanent_uuid: "7cfc9d0cbd72480995e129eac1dc993a" instance_seqno: 1790598228960882 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 12:23:49.182044 15578 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 12:23:49.182516 15809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82e827cf3dac4a34930c1e2e363897d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:23:49.182626 15809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82e827cf3dac4a34930c1e2e363897d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223b565174054d7fa706d88ce7edcdf6" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34799 } } peers { permanent_uuid: "7cfc9d0cbd72480995e129eac1dc993a" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 33199 } } peers { permanent_uuid: "82e827cf3dac4a34930c1e2e363897d6" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 46431 } }
I20260928 12:23:49.182912 15809 leader_election.cc:288] T 00000000000000000000000000000000 P 82e827cf3dac4a34930c1e2e363897d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 223b565174054d7fa706d88ce7edcdf6 (127.2.56.62:34799), 7cfc9d0cbd72480995e129eac1dc993a (127.2.56.61:33199)
I20260928 12:23:49.183291 15620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "82e827cf3dac4a34930c1e2e363897d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "223b565174054d7fa706d88ce7edcdf6" is_pre_election: true
I20260928 12:23:49.183429 15620 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 223b565174054d7fa706d88ce7edcdf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82e827cf3dac4a34930c1e2e363897d6 in term 0.
I20260928 12:23:49.183744 15745 leader_election.cc:302] T 00000000000000000000000000000000 P 82e827cf3dac4a34930c1e2e363897d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 223b565174054d7fa706d88ce7edcdf6, 82e827cf3dac4a34930c1e2e363897d6; no voters:
I20260928 12:23:49.183933 15809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82e827cf3dac4a34930c1e2e363897d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:23:49.183980 15809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82e827cf3dac4a34930c1e2e363897d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:23:49.184001 15809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82e827cf3dac4a34930c1e2e363897d6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:23:49.184896 15809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82e827cf3dac4a34930c1e2e363897d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223b565174054d7fa706d88ce7edcdf6" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34799 } } peers { permanent_uuid: "7cfc9d0cbd72480995e129eac1dc993a" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 33199 } } peers { permanent_uuid: "82e827cf3dac4a34930c1e2e363897d6" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 46431 } }
I20260928 12:23:49.185020 15809 leader_election.cc:288] T 00000000000000000000000000000000 P 82e827cf3dac4a34930c1e2e363897d6 [CANDIDATE]: Term 1 election: Requested vote from peers 223b565174054d7fa706d88ce7edcdf6 (127.2.56.62:34799), 7cfc9d0cbd72480995e129eac1dc993a (127.2.56.61:33199)
I20260928 12:23:49.185292 15620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "82e827cf3dac4a34930c1e2e363897d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "223b565174054d7fa706d88ce7edcdf6"
I20260928 12:23:49.185359 15620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 223b565174054d7fa706d88ce7edcdf6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:23:49.186134 15620 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 223b565174054d7fa706d88ce7edcdf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82e827cf3dac4a34930c1e2e363897d6 in term 1.
I20260928 12:23:49.186357 15745 leader_election.cc:302] T 00000000000000000000000000000000 P 82e827cf3dac4a34930c1e2e363897d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 223b565174054d7fa706d88ce7edcdf6, 82e827cf3dac4a34930c1e2e363897d6; no voters:
I20260928 12:23:49.186471 15809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82e827cf3dac4a34930c1e2e363897d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:23:49.186626 15809 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82e827cf3dac4a34930c1e2e363897d6 [term 1 LEADER]: Becoming Leader. State: Replica: 82e827cf3dac4a34930c1e2e363897d6, State: Running, Role: LEADER
I20260928 12:23:49.186705 15809 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82e827cf3dac4a34930c1e2e363897d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223b565174054d7fa706d88ce7edcdf6" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34799 } } peers { permanent_uuid: "7cfc9d0cbd72480995e129eac1dc993a" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 33199 } } peers { permanent_uuid: "82e827cf3dac4a34930c1e2e363897d6" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 46431 } }
I20260928 12:23:49.187135 15836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82e827cf3dac4a34930c1e2e363897d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82e827cf3dac4a34930c1e2e363897d6. Latest consensus state: current_term: 1 leader_uuid: "82e827cf3dac4a34930c1e2e363897d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223b565174054d7fa706d88ce7edcdf6" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34799 } } peers { permanent_uuid: "7cfc9d0cbd72480995e129eac1dc993a" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 33199 } } peers { permanent_uuid: "82e827cf3dac4a34930c1e2e363897d6" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 46431 } } }
I20260928 12:23:49.187507 15836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82e827cf3dac4a34930c1e2e363897d6 [sys.catalog]: This master's current role is: LEADER
I20260928 12:23:49.187917 15838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:23:49.188176 15838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:23:49.189605 15620 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 223b565174054d7fa706d88ce7edcdf6 [term 1 FOLLOWER]: Refusing update from remote peer 82e827cf3dac4a34930c1e2e363897d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:23:49.189913 15809 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 82e827cf3dac4a34930c1e2e363897d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "223b565174054d7fa706d88ce7edcdf6" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:23:49.191119 15823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 223b565174054d7fa706d88ce7edcdf6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82e827cf3dac4a34930c1e2e363897d6. Latest consensus state: current_term: 1 leader_uuid: "82e827cf3dac4a34930c1e2e363897d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223b565174054d7fa706d88ce7edcdf6" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34799 } } peers { permanent_uuid: "7cfc9d0cbd72480995e129eac1dc993a" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 33199 } } peers { permanent_uuid: "82e827cf3dac4a34930c1e2e363897d6" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 46431 } } }
I20260928 12:23:49.191217 15823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 223b565174054d7fa706d88ce7edcdf6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:23:49.191942 15809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82e827cf3dac4a34930c1e2e363897d6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "82e827cf3dac4a34930c1e2e363897d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223b565174054d7fa706d88ce7edcdf6" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34799 } } peers { permanent_uuid: "7cfc9d0cbd72480995e129eac1dc993a" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 33199 } } peers { permanent_uuid: "82e827cf3dac4a34930c1e2e363897d6" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 46431 } } }
I20260928 12:23:49.192019 15809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82e827cf3dac4a34930c1e2e363897d6 [sys.catalog]: This master's current role is: LEADER
I20260928 12:23:49.192385 15823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 223b565174054d7fa706d88ce7edcdf6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "82e827cf3dac4a34930c1e2e363897d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223b565174054d7fa706d88ce7edcdf6" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34799 } } peers { permanent_uuid: "7cfc9d0cbd72480995e129eac1dc993a" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 33199 } } peers { permanent_uuid: "82e827cf3dac4a34930c1e2e363897d6" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 46431 } } }
I20260928 12:23:49.192459 15823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 223b565174054d7fa706d88ce7edcdf6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:23:49.192514 15839 mvcc.cc:205] Tried to move back new op lower bound from 7334290346758447104 to 7334290346749337600. Current Snapshot: MvccSnapshot[applied={T|T < 7334290346758447104}]
I20260928 12:23:49.192629 15840 mvcc.cc:205] Tried to move back new op lower bound from 7334290346758447104 to 7334290346749337600. Current Snapshot: MvccSnapshot[applied={T|T < 7334290346758447104}]
I20260928 12:23:49.193470 15838 catalog_manager.cc:1347] Generated new cluster ID: 1115a392a83b468da165797aab2d2999
I20260928 12:23:49.193511 15838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:23:49.197295 15721 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.56.60" port: 46431 } attempt: 3
Sep 28 12:23:49 dist-test-slave-920p krb5kdc[15565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598228, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.61@KRBTEST.COM for kudu/127.2.56.60@KRBTEST.COM
I20260928 12:23:49.206349 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:49.197674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:46431 (local address 127.0.0.1:42194)
0928 12:23:49.197750 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:49.197761 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:49.197796 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:49.198234 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:49.198235 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:49.198429 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:23:49.198477 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:49.198478 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:49.199623 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:49.199624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:49.200456 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:49.200459 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:49.200949 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:49.200949 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:49.201003 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:49.201474 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:23:49.201479 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:23:49.203211 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:23:49.205055 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:49.205056 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:49.205056 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:49.205308 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:49.205568 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:49.205569 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:49.205570 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:49.205640 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:49.205980 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:23:49.205981 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:23:49.206046 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 12:23:49.206104 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 12:23:49.206210 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 12:23:49.206835 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:49.198005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42194 (local address 127.2.56.60:46431)
0928 12:23:49.198066 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:49.198068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:49.198074 (+ 6us) server_negotiation.cc:407] Connection header received
0928 12:23:49.198090 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:49.198091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:49.198100 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:49.198142 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:23:49.198555 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:49.199320 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:49.200548 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:49.200841 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:49.200898 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:49.203359 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:23:49.203361 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:23:49.203361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:23:49.203369 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:23:49.204940 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:49.205411 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:49.205412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:49.205412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:49.205487 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:49.205717 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:49.205718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:49.205718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:49.205877 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:23:49.205934 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:49.206575 (+ 641us) server_negotiation.cc:299] Negotiation successful
0928 12:23:49.206723 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 12:23:49.208427 15721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cfc9d0cbd72480995e129eac1dc993a: Bootstrap starting.
I20260928 12:23:49.209165 15721 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cfc9d0cbd72480995e129eac1dc993a: Neither blocks nor log segments found. Creating new log.
I20260928 12:23:49.209502 15721 log.cc:824] T 00000000000000000000000000000000 P 7cfc9d0cbd72480995e129eac1dc993a: Log is configured to *not* fsync() on all Append() calls
I20260928 12:23:49.211539 15721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cfc9d0cbd72480995e129eac1dc993a: No bootstrap required, opened a new log
I20260928 12:23:49.213593 15721 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cfc9d0cbd72480995e129eac1dc993a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223b565174054d7fa706d88ce7edcdf6" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34799 } } peers { permanent_uuid: "7cfc9d0cbd72480995e129eac1dc993a" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 33199 } } peers { permanent_uuid: "82e827cf3dac4a34930c1e2e363897d6" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 46431 } }
I20260928 12:23:49.213738 15721 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cfc9d0cbd72480995e129eac1dc993a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:23:49.213775 15721 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cfc9d0cbd72480995e129eac1dc993a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cfc9d0cbd72480995e129eac1dc993a, State: Initialized, Role: FOLLOWER
I20260928 12:23:49.213888 15721 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cfc9d0cbd72480995e129eac1dc993a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223b565174054d7fa706d88ce7edcdf6" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34799 } } peers { permanent_uuid: "7cfc9d0cbd72480995e129eac1dc993a" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 33199 } } peers { permanent_uuid: "82e827cf3dac4a34930c1e2e363897d6" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 46431 } }
I20260928 12:23:49.214207 15721 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cfc9d0cbd72480995e129eac1dc993a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:23:49.215585 15695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "82e827cf3dac4a34930c1e2e363897d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cfc9d0cbd72480995e129eac1dc993a" is_pre_election: true
I20260928 12:23:49.215678 15695 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7cfc9d0cbd72480995e129eac1dc993a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82e827cf3dac4a34930c1e2e363897d6 in term 0.
I20260928 12:23:49.215806 15693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cfc9d0cbd72480995e129eac1dc993a [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:23:49.215489 15694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "82e827cf3dac4a34930c1e2e363897d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cfc9d0cbd72480995e129eac1dc993a"
I20260928 12:23:49.217106 15693 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7cfc9d0cbd72480995e129eac1dc993a [term 1 FOLLOWER]: Refusing update from remote peer 82e827cf3dac4a34930c1e2e363897d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:23:49.217967 15836 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 82e827cf3dac4a34930c1e2e363897d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cfc9d0cbd72480995e129eac1dc993a" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 12:23:49.220423 15856 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7cfc9d0cbd72480995e129eac1dc993a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:23:49.220485 15856 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:23:49.220850 15858 mvcc.cc:205] Tried to move back new op lower bound from 7334290346758447104 to 7334290346749337600. Current Snapshot: MvccSnapshot[applied={T|T < 7334290346758447104}]
I20260928 12:23:49.221884 15860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cfc9d0cbd72480995e129eac1dc993a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82e827cf3dac4a34930c1e2e363897d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223b565174054d7fa706d88ce7edcdf6" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34799 } } peers { permanent_uuid: "7cfc9d0cbd72480995e129eac1dc993a" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 33199 } } peers { permanent_uuid: "82e827cf3dac4a34930c1e2e363897d6" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 46431 } } }
I20260928 12:23:49.221971 15860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cfc9d0cbd72480995e129eac1dc993a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:23:49.222019 15860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cfc9d0cbd72480995e129eac1dc993a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82e827cf3dac4a34930c1e2e363897d6. Latest consensus state: current_term: 1 leader_uuid: "82e827cf3dac4a34930c1e2e363897d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223b565174054d7fa706d88ce7edcdf6" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34799 } } peers { permanent_uuid: "7cfc9d0cbd72480995e129eac1dc993a" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 33199 } } peers { permanent_uuid: "82e827cf3dac4a34930c1e2e363897d6" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 46431 } } }
I20260928 12:23:49.222060 15860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cfc9d0cbd72480995e129eac1dc993a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:23:49.222084 15860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cfc9d0cbd72480995e129eac1dc993a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "82e827cf3dac4a34930c1e2e363897d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223b565174054d7fa706d88ce7edcdf6" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34799 } } peers { permanent_uuid: "7cfc9d0cbd72480995e129eac1dc993a" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 33199 } } peers { permanent_uuid: "82e827cf3dac4a34930c1e2e363897d6" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 46431 } } }
I20260928 12:23:49.222122 15860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cfc9d0cbd72480995e129eac1dc993a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:23:49.222812 15809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82e827cf3dac4a34930c1e2e363897d6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "82e827cf3dac4a34930c1e2e363897d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223b565174054d7fa706d88ce7edcdf6" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34799 } } peers { permanent_uuid: "7cfc9d0cbd72480995e129eac1dc993a" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 33199 } } peers { permanent_uuid: "82e827cf3dac4a34930c1e2e363897d6" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 46431 } } }
I20260928 12:23:49.222901 15809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82e827cf3dac4a34930c1e2e363897d6 [sys.catalog]: This master's current role is: LEADER
I20260928 12:23:49.222890 15838 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:23:49.223711 15838 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:23:49.230909 15838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82e827cf3dac4a34930c1e2e363897d6: Generated new TSK 0
I20260928 12:23:49.231127 15838 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:23:50.178822 15834 catalog_manager.cc:1259] Loaded cluster ID: 1115a392a83b468da165797aab2d2999
I20260928 12:23:50.178912 15834 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 223b565174054d7fa706d88ce7edcdf6: loading cluster ID for follower catalog manager: success
I20260928 12:23:50.179678 15834 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 223b565174054d7fa706d88ce7edcdf6: acquiring CA information for follower catalog manager: success
I20260928 12:23:50.179945 15834 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 223b565174054d7fa706d88ce7edcdf6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 12:23:50.220933 15856 catalog_manager.cc:1259] Loaded cluster ID: 1115a392a83b468da165797aab2d2999
I20260928 12:23:50.221014 15856 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7cfc9d0cbd72480995e129eac1dc993a: loading cluster ID for follower catalog manager: success
I20260928 12:23:50.221807 15856 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7cfc9d0cbd72480995e129eac1dc993a: acquiring CA information for follower catalog manager: success
I20260928 12:23:50.222088 15856 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7cfc9d0cbd72480995e129eac1dc993a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 12:23:50.286029 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 15578
I20260928 12:23:50.291044 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 15653
I20260928 12:23:50.296965 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 15728
2026-09-28T12:23:50Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2664 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:23:50 dist-test-slave-920p krb5kdc[15871](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:23:50 dist-test-slave-920p krb5kdc[15871](info): set up 2 sockets
Sep 28 12:23:50 dist-test-slave-920p krb5kdc[15871](info): commencing operation
krb5kdc: starting...
W20260928 12:23:51.269536 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:23:51 dist-test-slave-920p krb5kdc[15871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:23:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:23:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
I20260928 12:23:51.384861 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:39171
--webserver_interface=127.2.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:39355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:39171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:23:51.464164 15884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:23:51.464371 15884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:23:51.464393 15884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:23:51.465538 15884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:23:51.465574 15884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:23:51.465587 15884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:23:51.465600 15884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:23:51.465610 15884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:23:51.467049 15884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:39355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:39171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:39171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:23:51.467243 15884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:23:51.467408 15884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:23:51.471948 15889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:23:51.472101 15884 server_base.cc:1034] running on GCE node
W20260928 12:23:51.471978 15890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:23:51.471946 15892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:23:51.472381 15884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:23:51.472637 15884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:23:51.473776 15884 hybrid_clock.cc:648] HybridClock initialized: now 1790598231473764 us; error 27 us; skew 500 ppm
Sep 28 12:23:51 dist-test-slave-920p krb5kdc[15871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598231, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:23:51.482931 15884 init.cc:377] Logged in from keytab as kudu/127.2.56.62@KRBTEST.COM (short username kudu)
I20260928 12:23:51.483393 15884 webserver.cc:460] Webserver started at http://127.2.56.62:41345/ using document root <none> and password file <none>
I20260928 12:23:51.483598 15884 fs_manager.cc:359] Metadata directory not provided
I20260928 12:23:51.483645 15884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:23:51.483770 15884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:23:51.484798 15884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "b200dd7643544a108f84af586676b2cb"
format_stamp: "Formatted at 2026-09-28 12:23:51 on dist-test-slave-920p"
server_key: "1fc59a0e17155d30c98fb746dcbafa77"
server_key_iv: "5cabd6794f2ba7617e02f0f9758975ee"
server_key_version: "encryptionkey@0"
I20260928 12:23:51.485111 15884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "b200dd7643544a108f84af586676b2cb"
format_stamp: "Formatted at 2026-09-28 12:23:51 on dist-test-slave-920p"
server_key: "1fc59a0e17155d30c98fb746dcbafa77"
server_key_iv: "5cabd6794f2ba7617e02f0f9758975ee"
server_key_version: "encryptionkey@0"
I20260928 12:23:51.486388 15884 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:23:51.487074 15899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:51.487236 15884 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:23:51.487342 15884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "b200dd7643544a108f84af586676b2cb"
format_stamp: "Formatted at 2026-09-28 12:23:51 on dist-test-slave-920p"
server_key: "1fc59a0e17155d30c98fb746dcbafa77"
server_key_iv: "5cabd6794f2ba7617e02f0f9758975ee"
server_key_version: "encryptionkey@0"
I20260928 12:23:51.487407 15884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:23:51.500342 15884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:23:51.502362 15884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:23:51.502501 15884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:23:51.506285 15884 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:39171
I20260928 12:23:51.506372 15951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:39171 every 8 connection(s)
I20260928 12:23:51.506687 15884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:23:51.507503 15952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:23:51.509692 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 15884
I20260928 12:23:51.509783 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:23:51.509775 15952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b200dd7643544a108f84af586676b2cb: Bootstrap starting.
I20260928 12:23:51.509946 2272 external_mini_cluster.cc:1428] Setting key 35efb0243d3f771ae3a59d6cf690d05d
I20260928 12:23:51.510402 15952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b200dd7643544a108f84af586676b2cb: Neither blocks nor log segments found. Creating new log.
I20260928 12:23:51.510706 15952 log.cc:824] T 00000000000000000000000000000000 P b200dd7643544a108f84af586676b2cb: Log is configured to *not* fsync() on all Append() calls
I20260928 12:23:51.511415 15952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b200dd7643544a108f84af586676b2cb: No bootstrap required, opened a new log
I20260928 12:23:51.513646 15952 raft_consensus.cc:348] T 00000000000000000000000000000000 P b200dd7643544a108f84af586676b2cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b200dd7643544a108f84af586676b2cb" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39171 } }
I20260928 12:23:51.513784 15952 raft_consensus.cc:374] T 00000000000000000000000000000000 P b200dd7643544a108f84af586676b2cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:23:51.513805 15952 raft_consensus.cc:729] T 00000000000000000000000000000000 P b200dd7643544a108f84af586676b2cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b200dd7643544a108f84af586676b2cb, State: Initialized, Role: FOLLOWER
I20260928 12:23:51.513906 15952 consensus_queue.cc:260] T 00000000000000000000000000000000 P b200dd7643544a108f84af586676b2cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b200dd7643544a108f84af586676b2cb" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39171 } }
I20260928 12:23:51.513970 15952 raft_consensus.cc:388] T 00000000000000000000000000000000 P b200dd7643544a108f84af586676b2cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:23:51.513993 15952 raft_consensus.cc:482] T 00000000000000000000000000000000 P b200dd7643544a108f84af586676b2cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:23:51.514037 15952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b200dd7643544a108f84af586676b2cb [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:23:51.514670 15952 raft_consensus.cc:504] T 00000000000000000000000000000000 P b200dd7643544a108f84af586676b2cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b200dd7643544a108f84af586676b2cb" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39171 } }
I20260928 12:23:51.514776 15952 leader_election.cc:302] T 00000000000000000000000000000000 P b200dd7643544a108f84af586676b2cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b200dd7643544a108f84af586676b2cb; no voters:
I20260928 12:23:51.514942 15952 leader_election.cc:288] T 00000000000000000000000000000000 P b200dd7643544a108f84af586676b2cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:23:51.514981 15957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b200dd7643544a108f84af586676b2cb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:23:51.515146 15957 raft_consensus.cc:686] T 00000000000000000000000000000000 P b200dd7643544a108f84af586676b2cb [term 1 LEADER]: Becoming Leader. State: Replica: b200dd7643544a108f84af586676b2cb, State: Running, Role: LEADER
I20260928 12:23:51.515182 15952 sys_catalog.cc:564] T 00000000000000000000000000000000 P b200dd7643544a108f84af586676b2cb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:23:51.515230 15957 consensus_queue.cc:237] T 00000000000000000000000000000000 P b200dd7643544a108f84af586676b2cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b200dd7643544a108f84af586676b2cb" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39171 } }
I20260928 12:23:51.515575 15957 sys_catalog.cc:455] T 00000000000000000000000000000000 P b200dd7643544a108f84af586676b2cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b200dd7643544a108f84af586676b2cb. Latest consensus state: current_term: 1 leader_uuid: "b200dd7643544a108f84af586676b2cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b200dd7643544a108f84af586676b2cb" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39171 } } }
I20260928 12:23:51.515636 15957 sys_catalog.cc:458] T 00000000000000000000000000000000 P b200dd7643544a108f84af586676b2cb [sys.catalog]: This master's current role is: LEADER
I20260928 12:23:51.515784 15958 sys_catalog.cc:455] T 00000000000000000000000000000000 P b200dd7643544a108f84af586676b2cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b200dd7643544a108f84af586676b2cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b200dd7643544a108f84af586676b2cb" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39171 } } }
I20260928 12:23:51.515833 15958 sys_catalog.cc:458] T 00000000000000000000000000000000 P b200dd7643544a108f84af586676b2cb [sys.catalog]: This master's current role is: LEADER
Sep 28 12:23:51 dist-test-slave-920p krb5kdc[15871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
W20260928 12:23:51.516811 15973 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b200dd7643544a108f84af586676b2cb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:23:51.516860 15973 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:23:51.516899 15964 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:23:51.517048 15964 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:23:51.519532 15964 catalog_manager.cc:1347] Generated new cluster ID: f94fbd6eed484e6aaa1cde4e48b66462
I20260928 12:23:51.519599 15964 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:23:51.520682 15955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:51.510418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51204 (local address 127.2.56.62:39171)
0928 12:23:51.510583 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:51.510592 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:51.510629 (+ 37us) server_negotiation.cc:407] Connection header received
0928 12:23:51.511129 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:51.511137 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:51.511206 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:51.511358 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:23:51.511927 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:51.513319 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:51.514144 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:51.514571 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:51.514664 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:51.517327 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:23:51.517331 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:23:51.517332 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:23:51.517343 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:23:51.519123 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:51.519520 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:51.519523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:51.519523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:51.519635 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:51.519834 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:51.519835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:51.519835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:51.520116 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:23:51.520261 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:51.520364 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 12:23:51.520484 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
I20260928 12:23:51.521584 2272 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:23:51.528381 15964 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:23:51.528873 15964 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:23:51.534731 15964 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b200dd7643544a108f84af586676b2cb: Generated new TSK 0
I20260928 12:23:51.534914 15964 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:23:51.575474 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 15884
2026-09-28T12:23:51Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1276 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:23:51 dist-test-slave-920p krb5kdc[15979](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:23:51 dist-test-slave-920p krb5kdc[15979](info): set up 2 sockets
Sep 28 12:23:51 dist-test-slave-920p krb5kdc[15979](info): commencing operation
krb5kdc: starting...
W20260928 12:23:52.529232 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:23:52 dist-test-slave-920p krb5kdc[15979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:23:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:23:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
I20260928 12:23:52.649564 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:39297
--webserver_interface=127.2.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:43689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:39297
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:23:52.731819 15992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:23:52.732044 15992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:23:52.732066 15992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:23:52.733213 15992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:23:52.733256 15992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:23:52.733289 15992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:23:52.733302 15992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:23:52.733314 15992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:23:52.734887 15992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:43689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:39297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:39297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:23:52.735118 15992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:23:52.735320 15992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:23:52.740399 15992 server_base.cc:1034] running on GCE node
W20260928 12:23:52.740345 15998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:23:52.740343 15997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:23:52.740355 16000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:23:52.740796 15992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:23:52.741115 15992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:23:52.742285 15992 hybrid_clock.cc:648] HybridClock initialized: now 1790598232742248 us; error 46 us; skew 500 ppm
Sep 28 12:23:52 dist-test-slave-920p krb5kdc[15979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598232, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:23:52.744712 15992 init.cc:377] Logged in from keytab as kudu/127.2.56.62@KRBTEST.COM (short username kudu)
I20260928 12:23:52.745129 15992 webserver.cc:460] Webserver started at http://127.2.56.62:35425/ using document root <none> and password file <none>
I20260928 12:23:52.745385 15992 fs_manager.cc:359] Metadata directory not provided
I20260928 12:23:52.745440 15992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:23:52.745537 15992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:23:52.746593 15992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "edfd1d9e4b2845bd80adc01266c03757"
format_stamp: "Formatted at 2026-09-28 12:23:52 on dist-test-slave-920p"
server_key: "4180f1c2b871f2b365549cdd892b07f8"
server_key_iv: "57d546bbe2952e0e637b69c48fe6e1bf"
server_key_version: "encryptionkey@0"
I20260928 12:23:52.746896 15992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "edfd1d9e4b2845bd80adc01266c03757"
format_stamp: "Formatted at 2026-09-28 12:23:52 on dist-test-slave-920p"
server_key: "4180f1c2b871f2b365549cdd892b07f8"
server_key_iv: "57d546bbe2952e0e637b69c48fe6e1bf"
server_key_version: "encryptionkey@0"
I20260928 12:23:52.748230 15992 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:23:52.749087 16007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:52.749370 15992 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:23:52.749456 15992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "edfd1d9e4b2845bd80adc01266c03757"
format_stamp: "Formatted at 2026-09-28 12:23:52 on dist-test-slave-920p"
server_key: "4180f1c2b871f2b365549cdd892b07f8"
server_key_iv: "57d546bbe2952e0e637b69c48fe6e1bf"
server_key_version: "encryptionkey@0"
I20260928 12:23:52.749540 15992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:23:52.768433 15992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:23:52.770641 15992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:23:52.770780 15992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:23:52.774566 15992 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:39297
I20260928 12:23:52.774601 16059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:39297 every 8 connection(s)
I20260928 12:23:52.774947 15992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:23:52.776053 16060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:23:52.778501 16060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757: Bootstrap starting.
I20260928 12:23:52.779084 16060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757: Neither blocks nor log segments found. Creating new log.
I20260928 12:23:52.779486 16060 log.cc:824] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757: Log is configured to *not* fsync() on all Append() calls
I20260928 12:23:52.780305 16060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757: No bootstrap required, opened a new log
I20260928 12:23:52.782001 16060 raft_consensus.cc:348] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edfd1d9e4b2845bd80adc01266c03757" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39297 } }
I20260928 12:23:52.782145 16060 raft_consensus.cc:374] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:23:52.782172 16060 raft_consensus.cc:729] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edfd1d9e4b2845bd80adc01266c03757, State: Initialized, Role: FOLLOWER
I20260928 12:23:52.782292 16060 consensus_queue.cc:260] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edfd1d9e4b2845bd80adc01266c03757" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39297 } }
I20260928 12:23:52.782366 16060 raft_consensus.cc:388] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:23:52.782390 16060 raft_consensus.cc:482] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:23:52.782418 16060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:23:52.783209 16060 raft_consensus.cc:504] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edfd1d9e4b2845bd80adc01266c03757" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39297 } }
I20260928 12:23:52.783298 16060 leader_election.cc:302] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edfd1d9e4b2845bd80adc01266c03757; no voters:
I20260928 12:23:52.783428 16060 leader_election.cc:288] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:23:52.783489 16063 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:23:52.783655 16060 sys_catalog.cc:564] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:23:52.783658 16063 raft_consensus.cc:686] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [term 1 LEADER]: Becoming Leader. State: Replica: edfd1d9e4b2845bd80adc01266c03757, State: Running, Role: LEADER
I20260928 12:23:52.783830 16063 consensus_queue.cc:237] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edfd1d9e4b2845bd80adc01266c03757" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39297 } }
I20260928 12:23:52.784204 16064 sys_catalog.cc:455] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edfd1d9e4b2845bd80adc01266c03757" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edfd1d9e4b2845bd80adc01266c03757" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39297 } } }
I20260928 12:23:52.784283 16064 sys_catalog.cc:458] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [sys.catalog]: This master's current role is: LEADER
I20260928 12:23:52.784340 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 15992
I20260928 12:23:52.784433 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:23:52.784436 16065 sys_catalog.cc:455] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [sys.catalog]: SysCatalogTable state changed. Reason: New leader edfd1d9e4b2845bd80adc01266c03757. Latest consensus state: current_term: 1 leader_uuid: "edfd1d9e4b2845bd80adc01266c03757" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edfd1d9e4b2845bd80adc01266c03757" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39297 } } }
I20260928 12:23:52.784487 16065 sys_catalog.cc:458] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [sys.catalog]: This master's current role is: LEADER
I20260928 12:23:52.784533 16068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:23:52.784637 2272 external_mini_cluster.cc:1428] Setting key 6baadbe8925bd8994f7eb6f7a3012dd2
I20260928 12:23:52.785066 16068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:23:52.786784 16068 catalog_manager.cc:1347] Generated new cluster ID: a95b5298100148eca2ca97821ac794e3
I20260928 12:23:52.786830 16068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:23:52 dist-test-slave-920p krb5kdc[15979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:23:52.794294 16068 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:23:52.794818 16068 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:23:52.796682 16077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:52.785390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48738 (local address 127.2.56.62:39297)
0928 12:23:52.786012 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:52.786020 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:52.786045 (+ 25us) server_negotiation.cc:407] Connection header received
0928 12:23:52.786137 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:52.786142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:52.786186 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:52.786335 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:23:52.787039 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:52.788404 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:52.789536 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:52.790156 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:52.790261 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:52.793038 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:23:52.793043 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:23:52.793044 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:23:52.793057 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:23:52.795088 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:52.795482 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:52.795485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:52.795485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:52.795599 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:52.795767 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:52.795768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:52.795769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:52.796062 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:23:52.796212 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:52.796284 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 12:23:52.796433 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":84,"threads_started":1}
I20260928 12:23:52.798719 16068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757: Generated new TSK 0
I20260928 12:23:52.798962 16068 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab.
Entry for principal kudu/127.2.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab.
I20260928 12:23:52.832561 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.1:0
--local_ip_for_outbound_sockets=127.2.56.1
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:39297
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:43689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:23:52.912940 16086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:23:52.913159 16086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:23:52.913184 16086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:23:52.914426 16086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:23:52.914489 16086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:23:52.914518 16086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.1
I20260928 12:23:52.916146 16086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:43689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:39297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.1
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:23:52.916419 16086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:23:52.916637 16086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:23:52.922231 16091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:23:52.922334 16094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:23:52.922227 16092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:23:52.922605 16086 server_base.cc:1034] running on GCE node
I20260928 12:23:52.922927 16086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:23:52.923197 16086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:23:52.924381 16086 hybrid_clock.cc:648] HybridClock initialized: now 1790598232924362 us; error 32 us; skew 500 ppm
Sep 28 12:23:52 dist-test-slave-920p krb5kdc[15979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598232, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:23:52.927747 16086 init.cc:377] Logged in from keytab as kudu/127.2.56.1@KRBTEST.COM (short username kudu)
I20260928 12:23:52.928382 16086 webserver.cc:460] Webserver started at http://127.2.56.1:39933/ using document root <none> and password file <none>
I20260928 12:23:52.928619 16086 fs_manager.cc:359] Metadata directory not provided
I20260928 12:23:52.928673 16086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:23:52.928803 16086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:23:52.930019 16086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/data/instance:
uuid: "d798223e9d4c49b4b8dba3bd83defdcc"
format_stamp: "Formatted at 2026-09-28 12:23:52 on dist-test-slave-920p"
server_key: "5147b34ccacee2e75a5c1e03c3893046"
server_key_iv: "18f08ad9345d54a9d893e06ebce2539c"
server_key_version: "encryptionkey@0"
I20260928 12:23:52.930379 16086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance:
uuid: "d798223e9d4c49b4b8dba3bd83defdcc"
format_stamp: "Formatted at 2026-09-28 12:23:52 on dist-test-slave-920p"
server_key: "5147b34ccacee2e75a5c1e03c3893046"
server_key_iv: "18f08ad9345d54a9d893e06ebce2539c"
server_key_version: "encryptionkey@0"
I20260928 12:23:52.931625 16086 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:23:52.932325 16101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:52.932484 16086 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:23:52.932569 16086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/wal
uuid: "d798223e9d4c49b4b8dba3bd83defdcc"
format_stamp: "Formatted at 2026-09-28 12:23:52 on dist-test-slave-920p"
server_key: "5147b34ccacee2e75a5c1e03c3893046"
server_key_iv: "18f08ad9345d54a9d893e06ebce2539c"
server_key_version: "encryptionkey@0"
I20260928 12:23:52.932628 16086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:23:52.944809 16086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:23:52.946668 16086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:23:52.946806 16086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:23:52.947011 16086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:23:52.947296 16086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:23:52.947329 16086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:52.947359 16086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:23:52.947381 16086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:52.953464 16086 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.1:42581
I20260928 12:23:52.953516 16214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.1:42581 every 8 connection(s)
I20260928 12:23:52.953824 16086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
I20260928 12:23:52.957787 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 16086
I20260928 12:23:52.957937 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance
I20260928 12:23:52.958244 2272 external_mini_cluster.cc:1428] Setting key 7b6d9966e0e4c8cd70763429e9a31a6c
Sep 28 12:23:52 dist-test-slave-920p krb5kdc[15979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598232, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:23:52.963097 16077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:52.954484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:51905 (local address 127.2.56.62:39297)
0928 12:23:52.954557 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:52.954559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:52.954736 (+ 177us) server_negotiation.cc:407] Connection header received
0928 12:23:52.955213 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:52.955214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:52.955224 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:52.955305 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:23:52.956243 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:52.956792 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:52.958100 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:52.958318 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:52.958390 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:52.960522 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:23:52.960524 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:23:52.960524 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:23:52.960533 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:23:52.961799 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:52.962174 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:52.962175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:52.962176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:52.962228 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:52.962451 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:52.962451 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:52.962452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:52.962566 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:23:52.962622 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:52.962897 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 12:23:52.962999 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 12:23:52.963243 16217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:52.954463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.1:51905)
0928 12:23:52.954627 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:52.954665 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:52.955122 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:52.955369 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:52.955372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:52.955933 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:23:52.956125 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:52.956130 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:52.956901 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:52.956902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:52.957930 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:52.957933 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:52.958394 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:52.958395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:52.958519 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:52.959007 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:23:52.959016 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:23:52.960420 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:23:52.961875 (+ 1455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:52.961884 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:52.961886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:52.962107 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:52.962281 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:52.962282 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:52.962282 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:52.962396 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:52.962619 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:23:52.962625 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:23:52.962769 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 12:23:52.962857 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 12:23:52.963022 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":83,"threads_started":1}
I20260928 12:23:52.963964 16215 heartbeater.cc:347] Connected to a master server at 127.2.56.62:39297
I20260928 12:23:52.964071 16215 heartbeater.cc:464] Registering TS with master...
I20260928 12:23:52.964258 16215 heartbeater.cc:511] Master 127.2.56.62:39297 requested a full tablet report, sending...
I20260928 12:23:52.964879 16024 ts_manager.cc:194] Registered new tserver with Master: d798223e9d4c49b4b8dba3bd83defdcc (127.2.56.1:42581)
I20260928 12:23:52.965710 16024 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.1@KRBTEST.COM'} at 127.2.56.1:51905
WARNING: no policy specified for kudu/127.2.56.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab.
Entry for principal kudu/127.2.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab.
I20260928 12:23:52.988756 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.2:0
--local_ip_for_outbound_sockets=127.2.56.2
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:39297
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:43689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:23:53.071563 16220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:23:53.071782 16220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:23:53.071815 16220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:23:53.073019 16220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:23:53.073058 16220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:23:53.073096 16220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.2
I20260928 12:23:53.074589 16220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:43689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:39297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.2
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:23:53.074826 16220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:23:53.075033 16220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:23:53.079836 16226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:23:53.079944 16220 server_base.cc:1034] running on GCE node
W20260928 12:23:53.080006 16228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:23:53.079844 16225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:23:53.080304 16220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:23:53.080543 16220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:23:53.081732 16220 hybrid_clock.cc:648] HybridClock initialized: now 1790598233081713 us; error 34 us; skew 500 ppm
Sep 28 12:23:53 dist-test-slave-920p krb5kdc[15979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598233, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:23:53.084124 16220 init.cc:377] Logged in from keytab as kudu/127.2.56.2@KRBTEST.COM (short username kudu)
I20260928 12:23:53.084587 16220 webserver.cc:460] Webserver started at http://127.2.56.2:35281/ using document root <none> and password file <none>
I20260928 12:23:53.084815 16220 fs_manager.cc:359] Metadata directory not provided
I20260928 12:23:53.084859 16220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:23:53.084980 16220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:23:53.086056 16220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/data/instance:
uuid: "61304af90f2f400092fda93d54f75dc9"
format_stamp: "Formatted at 2026-09-28 12:23:53 on dist-test-slave-920p"
server_key: "bc731b2ecaa31fc5eccee0991f2645bc"
server_key_iv: "d94565324ec4f358b764ad68a6e25fdc"
server_key_version: "encryptionkey@0"
I20260928 12:23:53.086373 16220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance:
uuid: "61304af90f2f400092fda93d54f75dc9"
format_stamp: "Formatted at 2026-09-28 12:23:53 on dist-test-slave-920p"
server_key: "bc731b2ecaa31fc5eccee0991f2645bc"
server_key_iv: "d94565324ec4f358b764ad68a6e25fdc"
server_key_version: "encryptionkey@0"
I20260928 12:23:53.087581 16220 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 12:23:53.088274 16235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:53.088430 16220 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:23:53.088510 16220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/wal
uuid: "61304af90f2f400092fda93d54f75dc9"
format_stamp: "Formatted at 2026-09-28 12:23:53 on dist-test-slave-920p"
server_key: "bc731b2ecaa31fc5eccee0991f2645bc"
server_key_iv: "d94565324ec4f358b764ad68a6e25fdc"
server_key_version: "encryptionkey@0"
I20260928 12:23:53.088567 16220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:23:53.100737 16220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:23:53.102628 16220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:23:53.102767 16220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:23:53.102964 16220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:23:53.103240 16220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:23:53.103272 16220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:53.103302 16220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:23:53.103322 16220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:53.108547 16220 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.2:34035
I20260928 12:23:53.108590 16348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.2:34035 every 8 connection(s)
I20260928 12:23:53.108889 16220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
I20260928 12:23:53.113349 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 16220
I20260928 12:23:53.113447 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance
I20260928 12:23:53.113626 2272 external_mini_cluster.cc:1428] Setting key 96593104e08935efc6e4cab3350c6f96
Sep 28 12:23:53 dist-test-slave-920p krb5kdc[15979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598233, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:23:53.118090 16077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:53.109534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:41015 (local address 127.2.56.62:39297)
0928 12:23:53.109581 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:53.109583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:53.109766 (+ 183us) server_negotiation.cc:407] Connection header received
0928 12:23:53.110221 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:53.110222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:53.110232 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:53.110275 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:23:53.111160 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.111641 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.112651 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.112838 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.112898 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:53.115196 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:23:53.115198 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:23:53.115198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:23:53.115206 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:23:53.116623 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:53.117072 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:53.117073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:53.117073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:53.117132 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:53.117420 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:53.117420 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:53.117421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:53.117531 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:23:53.117585 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:53.117928 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 12:23:53.118020 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:23:53.118286 16351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:53.109511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.2:41015)
0928 12:23:53.109659 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:53.109692 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:53.110149 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:53.110336 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:53.110339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:53.110906 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:23:53.111098 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:53.111103 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.111738 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.111740 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:53.112568 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:53.112571 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.112906 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.112907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:53.113004 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:53.113609 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:23:53.113617 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:23:53.115105 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:23:53.116699 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:53.116706 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:53.116708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:53.116981 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:53.117205 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:53.117206 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:53.117206 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:53.117364 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:53.117588 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:23:53.117593 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:23:53.117763 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 12:23:53.117879 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 12:23:53.118058 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260928 12:23:53.118757 16349 heartbeater.cc:347] Connected to a master server at 127.2.56.62:39297
I20260928 12:23:53.118856 16349 heartbeater.cc:464] Registering TS with master...
I20260928 12:23:53.119057 16349 heartbeater.cc:511] Master 127.2.56.62:39297 requested a full tablet report, sending...
I20260928 12:23:53.119484 16024 ts_manager.cc:194] Registered new tserver with Master: 61304af90f2f400092fda93d54f75dc9 (127.2.56.2:34035)
I20260928 12:23:53.119864 16024 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.2@KRBTEST.COM'} at 127.2.56.2:41015
WARNING: no policy specified for kudu/127.2.56.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab.
Entry for principal kudu/127.2.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab.
I20260928 12:23:53.139942 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.3:0
--local_ip_for_outbound_sockets=127.2.56.3
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:39297
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:43689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:23:53.218518 16354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:23:53.218746 16354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:23:53.218780 16354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:23:53.219938 16354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:23:53.219980 16354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:23:53.220014 16354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.3
I20260928 12:23:53.221503 16354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:43689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:39297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.3
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:23:53.221751 16354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:23:53.221963 16354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:23:53.226950 16359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:23:53.226964 16362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:23:53.226960 16360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:23:53.227137 16354 server_base.cc:1034] running on GCE node
I20260928 12:23:53.227332 16354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:23:53.227566 16354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:23:53.228740 16354 hybrid_clock.cc:648] HybridClock initialized: now 1790598233228713 us; error 31 us; skew 500 ppm
Sep 28 12:23:53 dist-test-slave-920p krb5kdc[15979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598233, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:23:53.231200 16354 init.cc:377] Logged in from keytab as kudu/127.2.56.3@KRBTEST.COM (short username kudu)
I20260928 12:23:53.231717 16354 webserver.cc:460] Webserver started at http://127.2.56.3:36961/ using document root <none> and password file <none>
I20260928 12:23:53.231941 16354 fs_manager.cc:359] Metadata directory not provided
I20260928 12:23:53.231992 16354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:23:53.232120 16354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:23:53.233165 16354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/data/instance:
uuid: "979f8823ef2040a283f82ad07e1ed1ca"
format_stamp: "Formatted at 2026-09-28 12:23:53 on dist-test-slave-920p"
server_key: "d5a5dbf7f44c216c7970fc85674cdc47"
server_key_iv: "16dd5509fdf7c5e4c84d137b2fd3a782"
server_key_version: "encryptionkey@0"
I20260928 12:23:53.233552 16354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance:
uuid: "979f8823ef2040a283f82ad07e1ed1ca"
format_stamp: "Formatted at 2026-09-28 12:23:53 on dist-test-slave-920p"
server_key: "d5a5dbf7f44c216c7970fc85674cdc47"
server_key_iv: "16dd5509fdf7c5e4c84d137b2fd3a782"
server_key_version: "encryptionkey@0"
I20260928 12:23:53.234910 16354 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:23:53.235639 16369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:53.235805 16354 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:23:53.235886 16354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/wal
uuid: "979f8823ef2040a283f82ad07e1ed1ca"
format_stamp: "Formatted at 2026-09-28 12:23:53 on dist-test-slave-920p"
server_key: "d5a5dbf7f44c216c7970fc85674cdc47"
server_key_iv: "16dd5509fdf7c5e4c84d137b2fd3a782"
server_key_version: "encryptionkey@0"
I20260928 12:23:53.235941 16354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:23:53.243588 16354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:23:53.245406 16354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:23:53.245510 16354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:23:53.245672 16354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:23:53.245911 16354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:23:53.245939 16354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:53.245963 16354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:23:53.245980 16354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:53.251259 16354 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.3:45021
I20260928 12:23:53.251320 16482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.3:45021 every 8 connection(s)
I20260928 12:23:53.251576 16354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
I20260928 12:23:53.254231 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 16354
I20260928 12:23:53.254326 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance
I20260928 12:23:53.254500 2272 external_mini_cluster.cc:1428] Setting key ff8ff1ddde660b46535ad6af4d66f66d
Sep 28 12:23:53 dist-test-slave-920p krb5kdc[15979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598233, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:23:53.260269 16077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:53.252203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:33781 (local address 127.2.56.62:39297)
0928 12:23:53.252254 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:53.252256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:53.252468 (+ 212us) server_negotiation.cc:407] Connection header received
0928 12:23:53.253010 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:53.253011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:53.253023 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:53.253073 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:23:53.254089 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.254810 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.255920 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.256103 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.256151 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:53.258059 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:23:53.258060 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:23:53.258061 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:23:53.258067 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:23:53.259262 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:53.259585 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:53.259586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:53.259586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:53.259631 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:53.259816 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:53.259817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:53.259817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:53.259910 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:23:53.259953 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:53.260152 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 12:23:53.260207 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 12:23:53.260445 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:53.252180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.3:33781)
0928 12:23:53.252336 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:53.252389 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:53.252917 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:53.253141 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:53.253145 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:53.253791 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:23:53.253979 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:53.253985 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.254923 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.254924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:53.255839 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:53.255842 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.256160 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.256161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:53.256253 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:53.256663 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:23:53.256670 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:23:53.257976 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:23:53.259332 (+ 1356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:53.259337 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:53.259339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:53.259532 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:53.259673 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:53.259674 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:53.259674 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:53.259763 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:53.259951 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:23:53.259954 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:23:53.260061 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 12:23:53.260133 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 12:23:53.260242 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":63,"threads_started":1}
I20260928 12:23:53.260856 16483 heartbeater.cc:347] Connected to a master server at 127.2.56.62:39297
I20260928 12:23:53.260954 16483 heartbeater.cc:464] Registering TS with master...
I20260928 12:23:53.261119 16483 heartbeater.cc:511] Master 127.2.56.62:39297 requested a full tablet report, sending...
I20260928 12:23:53.261521 16024 ts_manager.cc:194] Registered new tserver with Master: 979f8823ef2040a283f82ad07e1ed1ca (127.2.56.3:45021)
I20260928 12:23:53.261888 16024 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.3@KRBTEST.COM'} at 127.2.56.3:33781
I20260928 12:23:53.266219 2272 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:23:53.273167 16077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:53.267630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48748 (local address 127.2.56.62:39297)
0928 12:23:53.267672 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:53.267674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:53.267775 (+ 101us) server_negotiation.cc:407] Connection header received
0928 12:23:53.267819 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:53.267820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:53.267828 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:53.267875 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:23:53.268398 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.268872 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.269614 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.269802 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.269853 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:53.270962 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:23:53.270963 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:23:53.270963 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:23:53.270969 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:23:53.272123 (+ 1154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:53.272432 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:53.272432 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:53.272433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:53.272477 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:53.272673 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:53.272673 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:53.272674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:53.272820 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:23:53.272893 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:53.273045 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 12:23:53.273102 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:23:53.275550 16024 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:23:53.278560 16024 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:23:53.285183 16491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:53.281037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:42581 (local address 127.0.0.1:59002)
0928 12:23:53.281206 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:53.281307 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:53.281374 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:53.281823 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:53.281824 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:53.281826 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:23:53.281872 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:53.281874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.283120 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.283121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:53.284074 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:53.284077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.284938 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.284939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:53.285002 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:53.285006 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:23:53.285057 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 12:23:53.285099 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":20,"thread_start_us":60,"threads_started":1}
I20260928 12:23:53.285194 16495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:53.281218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59002 (local address 127.2.56.1:42581)
0928 12:23:53.281605 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:53.281610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:53.281619 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:23:53.281656 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:53.281658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:53.281672 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:53.281748 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:23:53.281927 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.283005 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.284149 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.284867 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.284929 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:53.284984 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:53.285072 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 12:23:53.285110 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":100,"threads_started":1}
I20260928 12:23:53.285291 16493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:53.281151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37732 (local address 127.2.56.2:34035)
0928 12:23:53.281408 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:53.281413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:53.281482 (+ 69us) server_negotiation.cc:407] Connection header received
0928 12:23:53.281557 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:53.281559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:53.281575 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:53.281661 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:23:53.281836 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.282662 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.283973 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.284696 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.284740 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:53.284801 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:53.285144 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 12:23:53.285185 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":129,"threads_started":1}
I20260928 12:23:53.285581 16492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:53.281215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:45021 (local address 127.0.0.1:52908)
0928 12:23:53.281470 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:53.281478 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:53.281510 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:53.281998 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:53.281999 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:53.282001 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:23:53.282043 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:53.282045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.283292 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.283293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:53.284421 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:53.284424 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.285422 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.285423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:53.285473 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:53.285477 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:23:53.285507 (+ 30us) client_negotiation.cc:240] Negotiation successful
0928 12:23:53.285535 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":75,"threads_started":1}
I20260928 12:23:53.285688 16494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:53.281037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:34035 (local address 127.0.0.1:37732)
0928 12:23:53.281425 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:53.281435 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:53.281487 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:53.281668 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:53.281671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:53.281673 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:23:53.281751 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:53.281754 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.282752 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.282753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:53.283863 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:53.283866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.284856 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.284857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:53.284950 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:53.284962 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 12:23:53.285020 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 12:23:53.285069 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":4,"thread_start_us":56,"threads_started":1}
I20260928 12:23:53.286147 16496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:53.281425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52908 (local address 127.2.56.3:45021)
0928 12:23:53.281792 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:53.281795 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:53.281804 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:23:53.281838 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:53.281840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:53.281855 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:53.281923 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:23:53.282122 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.283175 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.284509 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.285297 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.285735 (+ 438us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:53.286003 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:53.286031 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 12:23:53.286071 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":98,"threads_started":1}
I20260928 12:23:53.286429 16149 tablet_service.cc:1467] Processing CreateTablet for tablet 4de49945e6a14dbdb767ef864fffd1d4 (DEFAULT_TABLE table=test-table [id=4ec1e236e9d74ea4937f82aacc0b3a92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:23:53.286499 16283 tablet_service.cc:1467] Processing CreateTablet for tablet 4de49945e6a14dbdb767ef864fffd1d4 (DEFAULT_TABLE table=test-table [id=4ec1e236e9d74ea4937f82aacc0b3a92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:23:53.286765 16283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4de49945e6a14dbdb767ef864fffd1d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:23:53.286770 16149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4de49945e6a14dbdb767ef864fffd1d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:23:53.286820 16417 tablet_service.cc:1467] Processing CreateTablet for tablet 4de49945e6a14dbdb767ef864fffd1d4 (DEFAULT_TABLE table=test-table [id=4ec1e236e9d74ea4937f82aacc0b3a92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:23:53.287079 16417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4de49945e6a14dbdb767ef864fffd1d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:23:53.290061 16497 tablet_bootstrap.cc:492] T 4de49945e6a14dbdb767ef864fffd1d4 P 61304af90f2f400092fda93d54f75dc9: Bootstrap starting.
I20260928 12:23:53.290766 16498 tablet_bootstrap.cc:492] T 4de49945e6a14dbdb767ef864fffd1d4 P 979f8823ef2040a283f82ad07e1ed1ca: Bootstrap starting.
I20260928 12:23:53.290946 16497 tablet_bootstrap.cc:654] T 4de49945e6a14dbdb767ef864fffd1d4 P 61304af90f2f400092fda93d54f75dc9: Neither blocks nor log segments found. Creating new log.
I20260928 12:23:53.291522 16498 tablet_bootstrap.cc:654] T 4de49945e6a14dbdb767ef864fffd1d4 P 979f8823ef2040a283f82ad07e1ed1ca: Neither blocks nor log segments found. Creating new log.
I20260928 12:23:53.291530 16497 log.cc:824] T 4de49945e6a14dbdb767ef864fffd1d4 P 61304af90f2f400092fda93d54f75dc9: Log is configured to *not* fsync() on all Append() calls
I20260928 12:23:53.292026 16498 log.cc:824] T 4de49945e6a14dbdb767ef864fffd1d4 P 979f8823ef2040a283f82ad07e1ed1ca: Log is configured to *not* fsync() on all Append() calls
I20260928 12:23:53.292335 16497 tablet_bootstrap.cc:492] T 4de49945e6a14dbdb767ef864fffd1d4 P 61304af90f2f400092fda93d54f75dc9: No bootstrap required, opened a new log
I20260928 12:23:53.292347 16499 tablet_bootstrap.cc:492] T 4de49945e6a14dbdb767ef864fffd1d4 P d798223e9d4c49b4b8dba3bd83defdcc: Bootstrap starting.
I20260928 12:23:53.292564 16497 ts_tablet_manager.cc:1397] T 4de49945e6a14dbdb767ef864fffd1d4 P 61304af90f2f400092fda93d54f75dc9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 12:23:53.292762 16498 tablet_bootstrap.cc:492] T 4de49945e6a14dbdb767ef864fffd1d4 P 979f8823ef2040a283f82ad07e1ed1ca: No bootstrap required, opened a new log
I20260928 12:23:53.292824 16498 ts_tablet_manager.cc:1397] T 4de49945e6a14dbdb767ef864fffd1d4 P 979f8823ef2040a283f82ad07e1ed1ca: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:23:53.292927 16499 tablet_bootstrap.cc:654] T 4de49945e6a14dbdb767ef864fffd1d4 P d798223e9d4c49b4b8dba3bd83defdcc: Neither blocks nor log segments found. Creating new log.
I20260928 12:23:53.293470 16499 log.cc:824] T 4de49945e6a14dbdb767ef864fffd1d4 P d798223e9d4c49b4b8dba3bd83defdcc: Log is configured to *not* fsync() on all Append() calls
I20260928 12:23:53.294178 16499 tablet_bootstrap.cc:492] T 4de49945e6a14dbdb767ef864fffd1d4 P d798223e9d4c49b4b8dba3bd83defdcc: No bootstrap required, opened a new log
I20260928 12:23:53.294251 16499 ts_tablet_manager.cc:1397] T 4de49945e6a14dbdb767ef864fffd1d4 P d798223e9d4c49b4b8dba3bd83defdcc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:23:53.294473 16498 raft_consensus.cc:348] T 4de49945e6a14dbdb767ef864fffd1d4 P 979f8823ef2040a283f82ad07e1ed1ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42581 } } peers { permanent_uuid: "61304af90f2f400092fda93d54f75dc9" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34035 } } peers { permanent_uuid: "979f8823ef2040a283f82ad07e1ed1ca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45021 } }
I20260928 12:23:53.294621 16498 raft_consensus.cc:374] T 4de49945e6a14dbdb767ef864fffd1d4 P 979f8823ef2040a283f82ad07e1ed1ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:23:53.294654 16498 raft_consensus.cc:729] T 4de49945e6a14dbdb767ef864fffd1d4 P 979f8823ef2040a283f82ad07e1ed1ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 979f8823ef2040a283f82ad07e1ed1ca, State: Initialized, Role: FOLLOWER
I20260928 12:23:53.294593 16497 raft_consensus.cc:348] T 4de49945e6a14dbdb767ef864fffd1d4 P 61304af90f2f400092fda93d54f75dc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42581 } } peers { permanent_uuid: "61304af90f2f400092fda93d54f75dc9" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34035 } } peers { permanent_uuid: "979f8823ef2040a283f82ad07e1ed1ca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45021 } }
I20260928 12:23:53.294785 16498 consensus_queue.cc:260] T 4de49945e6a14dbdb767ef864fffd1d4 P 979f8823ef2040a283f82ad07e1ed1ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42581 } } peers { permanent_uuid: "61304af90f2f400092fda93d54f75dc9" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34035 } } peers { permanent_uuid: "979f8823ef2040a283f82ad07e1ed1ca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45021 } }
I20260928 12:23:53.294844 16497 raft_consensus.cc:374] T 4de49945e6a14dbdb767ef864fffd1d4 P 61304af90f2f400092fda93d54f75dc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:23:53.294912 16497 raft_consensus.cc:729] T 4de49945e6a14dbdb767ef864fffd1d4 P 61304af90f2f400092fda93d54f75dc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61304af90f2f400092fda93d54f75dc9, State: Initialized, Role: FOLLOWER
I20260928 12:23:53.295068 16498 ts_tablet_manager.cc:1428] T 4de49945e6a14dbdb767ef864fffd1d4 P 979f8823ef2040a283f82ad07e1ed1ca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:23:53.295056 16497 consensus_queue.cc:260] T 4de49945e6a14dbdb767ef864fffd1d4 P 61304af90f2f400092fda93d54f75dc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42581 } } peers { permanent_uuid: "61304af90f2f400092fda93d54f75dc9" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34035 } } peers { permanent_uuid: "979f8823ef2040a283f82ad07e1ed1ca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45021 } }
I20260928 12:23:53.295089 16483 heartbeater.cc:503] Master 127.2.56.62:39297 was elected leader, sending a full tablet report...
I20260928 12:23:53.295315 16349 heartbeater.cc:503] Master 127.2.56.62:39297 was elected leader, sending a full tablet report...
I20260928 12:23:53.295766 16497 ts_tablet_manager.cc:1428] T 4de49945e6a14dbdb767ef864fffd1d4 P 61304af90f2f400092fda93d54f75dc9: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260928 12:23:53.296206 16499 raft_consensus.cc:348] T 4de49945e6a14dbdb767ef864fffd1d4 P d798223e9d4c49b4b8dba3bd83defdcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42581 } } peers { permanent_uuid: "61304af90f2f400092fda93d54f75dc9" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34035 } } peers { permanent_uuid: "979f8823ef2040a283f82ad07e1ed1ca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45021 } }
I20260928 12:23:53.296383 16499 raft_consensus.cc:374] T 4de49945e6a14dbdb767ef864fffd1d4 P d798223e9d4c49b4b8dba3bd83defdcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:23:53.296429 16499 raft_consensus.cc:729] T 4de49945e6a14dbdb767ef864fffd1d4 P d798223e9d4c49b4b8dba3bd83defdcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d798223e9d4c49b4b8dba3bd83defdcc, State: Initialized, Role: FOLLOWER
I20260928 12:23:53.296531 16499 consensus_queue.cc:260] T 4de49945e6a14dbdb767ef864fffd1d4 P d798223e9d4c49b4b8dba3bd83defdcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42581 } } peers { permanent_uuid: "61304af90f2f400092fda93d54f75dc9" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34035 } } peers { permanent_uuid: "979f8823ef2040a283f82ad07e1ed1ca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45021 } }
I20260928 12:23:53.296785 16499 ts_tablet_manager.cc:1428] T 4de49945e6a14dbdb767ef864fffd1d4 P d798223e9d4c49b4b8dba3bd83defdcc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:23:53.296800 16215 heartbeater.cc:503] Master 127.2.56.62:39297 was elected leader, sending a full tablet report...
I20260928 12:23:53.326249 16503 raft_consensus.cc:482] T 4de49945e6a14dbdb767ef864fffd1d4 P 979f8823ef2040a283f82ad07e1ed1ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:23:53.326409 16503 raft_consensus.cc:504] T 4de49945e6a14dbdb767ef864fffd1d4 P 979f8823ef2040a283f82ad07e1ed1ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42581 } } peers { permanent_uuid: "61304af90f2f400092fda93d54f75dc9" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34035 } } peers { permanent_uuid: "979f8823ef2040a283f82ad07e1ed1ca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45021 } }
I20260928 12:23:53.326817 16503 leader_election.cc:288] T 4de49945e6a14dbdb767ef864fffd1d4 P 979f8823ef2040a283f82ad07e1ed1ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d798223e9d4c49b4b8dba3bd83defdcc (127.2.56.1:42581), 61304af90f2f400092fda93d54f75dc9 (127.2.56.2:34035)
I20260928 12:23:53.330453 16495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:53.327025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:52203 (local address 127.2.56.1:42581)
0928 12:23:53.327064 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:53.327065 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:53.327198 (+ 133us) server_negotiation.cc:407] Connection header received
0928 12:23:53.327245 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:53.327246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:53.327256 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:53.327293 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:23:53.327426 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.328108 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.329523 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.330098 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.330166 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:53.330175 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:53.330337 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 12:23:53.330372 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:23:53.330446 16493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:53.327036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:51793 (local address 127.2.56.2:34035)
0928 12:23:53.327081 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:53.327083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:53.327092 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:23:53.327119 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:53.327120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:53.327131 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:53.327188 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:23:53.327355 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.328109 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.329527 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.330097 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.330164 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:53.330175 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:53.330334 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 12:23:53.330371 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:23:53.330451 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:53.326959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:34035 (local address 127.2.56.3:51793)
0928 12:23:53.327013 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:53.327025 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:53.327061 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:53.327203 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:53.327204 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:53.327206 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:23:53.327275 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:53.327277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.328180 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.328181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:53.329445 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:53.329448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.330175 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.330176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:53.330261 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:53.330270 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 12:23:53.330317 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 12:23:53.330362 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":21}
I20260928 12:23:53.330461 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:53.326964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:42581 (local address 127.2.56.3:52203)
0928 12:23:53.327140 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:53.327149 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:53.327198 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:53.327307 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:53.327308 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:53.327309 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:23:53.327365 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:53.327366 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.328175 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.328176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:53.329445 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:53.329448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.330175 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.330176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:53.330262 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:53.330270 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 12:23:53.330317 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 12:23:53.330362 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":74,"thread_start_us":117,"threads_started":1}
I20260928 12:23:53.330812 16303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4de49945e6a14dbdb767ef864fffd1d4" candidate_uuid: "979f8823ef2040a283f82ad07e1ed1ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61304af90f2f400092fda93d54f75dc9" is_pre_election: true
I20260928 12:23:53.330813 16169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4de49945e6a14dbdb767ef864fffd1d4" candidate_uuid: "979f8823ef2040a283f82ad07e1ed1ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" is_pre_election: true
I20260928 12:23:53.330972 16169 raft_consensus.cc:2454] T 4de49945e6a14dbdb767ef864fffd1d4 P d798223e9d4c49b4b8dba3bd83defdcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 979f8823ef2040a283f82ad07e1ed1ca in term 0.
I20260928 12:23:53.330972 16303 raft_consensus.cc:2454] T 4de49945e6a14dbdb767ef864fffd1d4 P 61304af90f2f400092fda93d54f75dc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 979f8823ef2040a283f82ad07e1ed1ca in term 0.
I20260928 12:23:53.331178 16373 leader_election.cc:302] T 4de49945e6a14dbdb767ef864fffd1d4 P 979f8823ef2040a283f82ad07e1ed1ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 979f8823ef2040a283f82ad07e1ed1ca, d798223e9d4c49b4b8dba3bd83defdcc; no voters:
I20260928 12:23:53.331310 16503 raft_consensus.cc:2781] T 4de49945e6a14dbdb767ef864fffd1d4 P 979f8823ef2040a283f82ad07e1ed1ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:23:53.331363 16503 raft_consensus.cc:482] T 4de49945e6a14dbdb767ef864fffd1d4 P 979f8823ef2040a283f82ad07e1ed1ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:23:53.331386 16503 raft_consensus.cc:3037] T 4de49945e6a14dbdb767ef864fffd1d4 P 979f8823ef2040a283f82ad07e1ed1ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:23:53.332043 16503 raft_consensus.cc:504] T 4de49945e6a14dbdb767ef864fffd1d4 P 979f8823ef2040a283f82ad07e1ed1ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42581 } } peers { permanent_uuid: "61304af90f2f400092fda93d54f75dc9" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34035 } } peers { permanent_uuid: "979f8823ef2040a283f82ad07e1ed1ca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45021 } }
I20260928 12:23:53.332157 16503 leader_election.cc:288] T 4de49945e6a14dbdb767ef864fffd1d4 P 979f8823ef2040a283f82ad07e1ed1ca [CANDIDATE]: Term 1 election: Requested vote from peers d798223e9d4c49b4b8dba3bd83defdcc (127.2.56.1:42581), 61304af90f2f400092fda93d54f75dc9 (127.2.56.2:34035)
I20260928 12:23:53.332329 16169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4de49945e6a14dbdb767ef864fffd1d4" candidate_uuid: "979f8823ef2040a283f82ad07e1ed1ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d798223e9d4c49b4b8dba3bd83defdcc"
I20260928 12:23:53.332401 16169 raft_consensus.cc:3037] T 4de49945e6a14dbdb767ef864fffd1d4 P d798223e9d4c49b4b8dba3bd83defdcc [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:23:53.333047 16169 raft_consensus.cc:2454] T 4de49945e6a14dbdb767ef864fffd1d4 P d798223e9d4c49b4b8dba3bd83defdcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 979f8823ef2040a283f82ad07e1ed1ca in term 1.
I20260928 12:23:53.333220 16373 leader_election.cc:302] T 4de49945e6a14dbdb767ef864fffd1d4 P 979f8823ef2040a283f82ad07e1ed1ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 979f8823ef2040a283f82ad07e1ed1ca, d798223e9d4c49b4b8dba3bd83defdcc; no voters:
I20260928 12:23:53.333377 16503 raft_consensus.cc:2781] T 4de49945e6a14dbdb767ef864fffd1d4 P 979f8823ef2040a283f82ad07e1ed1ca [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:23:53.333532 16503 raft_consensus.cc:686] T 4de49945e6a14dbdb767ef864fffd1d4 P 979f8823ef2040a283f82ad07e1ed1ca [term 1 LEADER]: Becoming Leader. State: Replica: 979f8823ef2040a283f82ad07e1ed1ca, State: Running, Role: LEADER
I20260928 12:23:53.333149 16303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4de49945e6a14dbdb767ef864fffd1d4" candidate_uuid: "979f8823ef2040a283f82ad07e1ed1ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61304af90f2f400092fda93d54f75dc9"
I20260928 12:23:53.333608 16303 raft_consensus.cc:3037] T 4de49945e6a14dbdb767ef864fffd1d4 P 61304af90f2f400092fda93d54f75dc9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:23:53.333629 16503 consensus_queue.cc:237] T 4de49945e6a14dbdb767ef864fffd1d4 P 979f8823ef2040a283f82ad07e1ed1ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42581 } } peers { permanent_uuid: "61304af90f2f400092fda93d54f75dc9" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34035 } } peers { permanent_uuid: "979f8823ef2040a283f82ad07e1ed1ca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45021 } }
I20260928 12:23:53.334635 16303 raft_consensus.cc:2454] T 4de49945e6a14dbdb767ef864fffd1d4 P 61304af90f2f400092fda93d54f75dc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 979f8823ef2040a283f82ad07e1ed1ca in term 1.
I20260928 12:23:53.334836 16024 catalog_manager.cc:5495] T 4de49945e6a14dbdb767ef864fffd1d4 P 979f8823ef2040a283f82ad07e1ed1ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 979f8823ef2040a283f82ad07e1ed1ca (127.2.56.3). New cstate: current_term: 1 leader_uuid: "979f8823ef2040a283f82ad07e1ed1ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61304af90f2f400092fda93d54f75dc9" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "979f8823ef2040a283f82ad07e1ed1ca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45021 } health_report { overall_health: HEALTHY } } }
I20260928 12:23:53.342708 16496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:53.340484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52918 (local address 127.2.56.3:45021)
0928 12:23:53.340558 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:53.340560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:53.340569 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:23:53.340596 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:53.340597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:53.340612 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:53.340676 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:23:53.340838 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.341366 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.342090 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.342266 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.342305 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:53.342462 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:23:53.342553 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:23:53.342606 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:53.342625 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 12:23:53.342640 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 12:23:53.344272 16303 raft_consensus.cc:1261] T 4de49945e6a14dbdb767ef864fffd1d4 P 61304af90f2f400092fda93d54f75dc9 [term 1 FOLLOWER]: Refusing update from remote peer 979f8823ef2040a283f82ad07e1ed1ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:23:53.344331 16169 raft_consensus.cc:1261] T 4de49945e6a14dbdb767ef864fffd1d4 P d798223e9d4c49b4b8dba3bd83defdcc [term 1 FOLLOWER]: Refusing update from remote peer 979f8823ef2040a283f82ad07e1ed1ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:23:53.344579 16507 consensus_queue.cc:1035] T 4de49945e6a14dbdb767ef864fffd1d4 P 979f8823ef2040a283f82ad07e1ed1ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "61304af90f2f400092fda93d54f75dc9" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:23:53.344712 16503 consensus_queue.cc:1035] T 4de49945e6a14dbdb767ef864fffd1d4 P 979f8823ef2040a283f82ad07e1ed1ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:23:53.346891 16510 mvcc.cc:205] Tried to move back new op lower bound from 7334290363775946752 to 7334290363735273472. Current Snapshot: MvccSnapshot[applied={T|T < 7334290363775946752}]
I20260928 12:23:53.347307 16511 mvcc.cc:205] Tried to move back new op lower bound from 7334290363775946752 to 7334290363735273472. Current Snapshot: MvccSnapshot[applied={T|T < 7334290363775946752}]
I20260928 12:23:53.349074 16024 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:23:53.349149 16024 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:23:53.349828 16024 catalog_manager.cc:5782] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757: Sending DeleteTablet for 3 replicas of tablet 4de49945e6a14dbdb767ef864fffd1d4
I20260928 12:23:53.350127 16417 tablet_service.cc:1514] Processing DeleteTablet for tablet 4de49945e6a14dbdb767ef864fffd1d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:23:53 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:52908
I20260928 12:23:53.350135 16283 tablet_service.cc:1514] Processing DeleteTablet for tablet 4de49945e6a14dbdb767ef864fffd1d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:23:53 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:37732
I20260928 12:23:53.350168 16149 tablet_service.cc:1514] Processing DeleteTablet for tablet 4de49945e6a14dbdb767ef864fffd1d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:23:53 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:59002
I20260928 12:23:53.350364 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 16086
I20260928 12:23:53.350433 16520 tablet_replica.cc:330] T 4de49945e6a14dbdb767ef864fffd1d4 P 979f8823ef2040a283f82ad07e1ed1ca: stopping tablet replica
I20260928 12:23:53.350553 16520 raft_consensus.cc:2229] T 4de49945e6a14dbdb767ef864fffd1d4 P 979f8823ef2040a283f82ad07e1ed1ca [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:23:53.351132 16520 raft_consensus.cc:2258] T 4de49945e6a14dbdb767ef864fffd1d4 P 979f8823ef2040a283f82ad07e1ed1ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:23:53.351624 16520 ts_tablet_manager.cc:1905] T 4de49945e6a14dbdb767ef864fffd1d4 P 979f8823ef2040a283f82ad07e1ed1ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:23:53.353811 16520 ts_tablet_manager.cc:1918] T 4de49945e6a14dbdb767ef864fffd1d4 P 979f8823ef2040a283f82ad07e1ed1ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:23:53.353879 16520 log.cc:1196] T 4de49945e6a14dbdb767ef864fffd1d4 P 979f8823ef2040a283f82ad07e1ed1ca: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/wal/wals/4de49945e6a14dbdb767ef864fffd1d4
I20260928 12:23:53.354097 16520 ts_tablet_manager.cc:1939] T 4de49945e6a14dbdb767ef864fffd1d4 P 979f8823ef2040a283f82ad07e1ed1ca: Deleting consensus metadata
I20260928 12:23:53.354461 16011 catalog_manager.cc:4841] TS 979f8823ef2040a283f82ad07e1ed1ca (127.2.56.3:45021): tablet 4de49945e6a14dbdb767ef864fffd1d4 (table test-table [id=4ec1e236e9d74ea4937f82aacc0b3a92]) successfully deleted
W20260928 12:23:53.357337 16011 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.56.1:42581 (error 108)
I20260928 12:23:53.357554 16522 tablet_replica.cc:330] T 4de49945e6a14dbdb767ef864fffd1d4 P 61304af90f2f400092fda93d54f75dc9: stopping tablet replica
I20260928 12:23:53.357690 16522 raft_consensus.cc:2229] T 4de49945e6a14dbdb767ef864fffd1d4 P 61304af90f2f400092fda93d54f75dc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:23:53.357803 16494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:53.357681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:42581 (local address 127.0.0.1:59006)
0928 12:23:53.357723 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:53.357768 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.1:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260928 12:23:53.357811 16522 raft_consensus.cc:2258] T 4de49945e6a14dbdb767ef864fffd1d4 P 61304af90f2f400092fda93d54f75dc9 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 12:23:53.357956 16011 catalog_manager.cc:4584] TS d798223e9d4c49b4b8dba3bd83defdcc (127.2.56.1:42581): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4de49945e6a14dbdb767ef864fffd1d4: Network error: Client connection negotiation failed: client connection to 127.2.56.1:42581: connect: Connection refused (error 111)
I20260928 12:23:53.358177 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 16220
I20260928 12:23:53.358338 16522 ts_tablet_manager.cc:1905] T 4de49945e6a14dbdb767ef864fffd1d4 P 61304af90f2f400092fda93d54f75dc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:23:53.363530 16494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:53.363397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:34035 (local address 127.0.0.1:37740)
0928 12:23:53.363456 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:53.363488 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.2:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 12:23:53.363528 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 16354
I20260928 12:23:53.367799 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 15992
I20260928 12:23:53.372326 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:39297
--webserver_interface=127.2.56.62
--webserver_port=35425
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:43689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:39297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:23:53.453977 16524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:23:53.454180 16524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:23:53.454200 16524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:23:53.455336 16524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:23:53.455382 16524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:23:53.455396 16524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:23:53.455407 16524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:23:53.455417 16524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:23:53.456875 16524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:43689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:39297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:39297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=35425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:23:53.457106 16524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:23:53.457329 16524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:23:53.465322 16531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:23:53.465337 16533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:23:53.465337 16530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:23:53.465458 16524 server_base.cc:1034] running on GCE node
I20260928 12:23:53.465801 16524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:23:53.466125 16524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:23:53.467260 16524 hybrid_clock.cc:648] HybridClock initialized: now 1790598233467241 us; error 34 us; skew 500 ppm
Sep 28 12:23:53 dist-test-slave-920p krb5kdc[15979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598233, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:23:53.469696 16524 init.cc:377] Logged in from keytab as kudu/127.2.56.62@KRBTEST.COM (short username kudu)
I20260928 12:23:53.470151 16524 webserver.cc:460] Webserver started at http://127.2.56.62:35425/ using document root <none> and password file <none>
I20260928 12:23:53.470367 16524 fs_manager.cc:359] Metadata directory not provided
I20260928 12:23:53.470418 16524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:23:53.471663 16524 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:23:53.472225 16540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:53.472365 16524 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:53.472447 16524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "edfd1d9e4b2845bd80adc01266c03757"
format_stamp: "Formatted at 2026-09-28 12:23:52 on dist-test-slave-920p"
server_key: "4180f1c2b871f2b365549cdd892b07f8"
server_key_iv: "57d546bbe2952e0e637b69c48fe6e1bf"
server_key_version: "encryptionkey@0"
I20260928 12:23:53.472832 16524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:23:53.487807 16524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:23:53.489854 16524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:23:53.489985 16524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:23:53.493669 16524 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:39297
I20260928 12:23:53.493702 16592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:39297 every 8 connection(s)
I20260928 12:23:53.494058 16524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:23:53.495579 16593 sys_catalog.cc:263] Verifying existing consensus state
I20260928 12:23:53.496215 16593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757: Bootstrap starting.
I20260928 12:23:53.497013 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 16524
I20260928 12:23:53.497323 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.1:42581
--local_ip_for_outbound_sockets=127.2.56.1
--tserver_master_addrs=127.2.56.62:39297
--webserver_port=39933
--webserver_interface=127.2.56.1
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:43689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:23:53.504841 16593 log.cc:824] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757: Log is configured to *not* fsync() on all Append() calls
I20260928 12:23:53.506990 16593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:23:53.507241 16593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757: Bootstrap complete.
I20260928 12:23:53.509421 16593 raft_consensus.cc:348] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edfd1d9e4b2845bd80adc01266c03757" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39297 } }
I20260928 12:23:53.509657 16593 raft_consensus.cc:729] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: edfd1d9e4b2845bd80adc01266c03757, State: Initialized, Role: FOLLOWER
I20260928 12:23:53.509804 16593 consensus_queue.cc:260] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edfd1d9e4b2845bd80adc01266c03757" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39297 } }
I20260928 12:23:53.509860 16593 raft_consensus.cc:388] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:23:53.509905 16593 raft_consensus.cc:482] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:23:53.509950 16593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [term 1 FOLLOWER]: Advancing to term 2
I20260928 12:23:53.510941 16593 raft_consensus.cc:504] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edfd1d9e4b2845bd80adc01266c03757" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39297 } }
I20260928 12:23:53.511025 16593 leader_election.cc:302] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edfd1d9e4b2845bd80adc01266c03757; no voters:
I20260928 12:23:53.511189 16593 leader_election.cc:288] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 12:23:53.511231 16597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 12:23:53.511386 16597 raft_consensus.cc:686] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [term 2 LEADER]: Becoming Leader. State: Replica: edfd1d9e4b2845bd80adc01266c03757, State: Running, Role: LEADER
I20260928 12:23:53.511451 16593 sys_catalog.cc:564] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:23:53.511531 16597 consensus_queue.cc:237] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edfd1d9e4b2845bd80adc01266c03757" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39297 } }
I20260928 12:23:53.511788 16597 sys_catalog.cc:455] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [sys.catalog]: SysCatalogTable state changed. Reason: New leader edfd1d9e4b2845bd80adc01266c03757. Latest consensus state: current_term: 2 leader_uuid: "edfd1d9e4b2845bd80adc01266c03757" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edfd1d9e4b2845bd80adc01266c03757" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39297 } } }
I20260928 12:23:53.512035 16597 sys_catalog.cc:458] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [sys.catalog]: This master's current role is: LEADER
I20260928 12:23:53.511967 16599 sys_catalog.cc:455] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "edfd1d9e4b2845bd80adc01266c03757" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edfd1d9e4b2845bd80adc01266c03757" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39297 } } }
I20260928 12:23:53.512391 16599 sys_catalog.cc:458] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [sys.catalog]: This master's current role is: LEADER
I20260928 12:23:53.512481 16606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:23:53.513252 16606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:23:53.513422 16606 catalog_manager.cc:1259] Loaded cluster ID: a95b5298100148eca2ca97821ac794e3
I20260928 12:23:53.513674 16606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:23:53.513944 16606 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 12:23:53.518635 16606 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 12:23:53.521054 16606 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:23:53.521258 16606 catalog_manager.cc:5879] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757: Loaded TSK: 0
I20260928 12:23:53.521513 16606 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 12:23:53.579927 16595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:23:53.580133 16595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:23:53.580157 16595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:23:53.581369 16595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:23:53.581437 16595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:23:53.581475 16595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.1
I20260928 12:23:53.582937 16595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:43689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.1:42581
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.56.1
--webserver_port=39933
--tserver_master_addrs=127.2.56.62:39297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.1
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:23:53.583150 16595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:23:53.583361 16595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:23:53.589083 16620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:23:53.589119 16621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:23:53.589077 16623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:23:53.589529 16595 server_base.cc:1034] running on GCE node
I20260928 12:23:53.589716 16595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:23:53.589939 16595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:23:53.591094 16595 hybrid_clock.cc:648] HybridClock initialized: now 1790598233591078 us; error 31 us; skew 500 ppm
Sep 28 12:23:53 dist-test-slave-920p krb5kdc[15979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598233, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:23:53.593436 16595 init.cc:377] Logged in from keytab as kudu/127.2.56.1@KRBTEST.COM (short username kudu)
I20260928 12:23:53.593858 16595 webserver.cc:460] Webserver started at http://127.2.56.1:39933/ using document root <none> and password file <none>
I20260928 12:23:53.594074 16595 fs_manager.cc:359] Metadata directory not provided
I20260928 12:23:53.594135 16595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:23:53.595510 16595 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:23:53.596139 16630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:53.596304 16595 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:23:53.596410 16595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/wal
uuid: "d798223e9d4c49b4b8dba3bd83defdcc"
format_stamp: "Formatted at 2026-09-28 12:23:52 on dist-test-slave-920p"
server_key: "5147b34ccacee2e75a5c1e03c3893046"
server_key_iv: "18f08ad9345d54a9d893e06ebce2539c"
server_key_version: "encryptionkey@0"
I20260928 12:23:53.596853 16595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:23:53.607322 16595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:23:53.609865 16595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:23:53.610019 16595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:23:53.610284 16595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:23:53.610783 16637 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 12:23:53.612439 16595 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 12:23:53.612501 16595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 12:23:53.612533 16595 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 12:23:53.613493 16595 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 12:23:53.613541 16595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260928 12:23:53.613596 16637 tablet_bootstrap.cc:492] T 4de49945e6a14dbdb767ef864fffd1d4 P d798223e9d4c49b4b8dba3bd83defdcc: Bootstrap starting.
I20260928 12:23:53.620524 16595 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.1:42581
I20260928 12:23:53.620678 16744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.1:42581 every 8 connection(s)
I20260928 12:23:53.621032 16595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
I20260928 12:23:53.622306 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 16595
I20260928 12:23:53.622582 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.2:34035
--local_ip_for_outbound_sockets=127.2.56.2
--tserver_master_addrs=127.2.56.62:39297
--webserver_port=35281
--webserver_interface=127.2.56.2
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:43689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:23:53.623567 16637 log.cc:824] T 4de49945e6a14dbdb767ef864fffd1d4 P d798223e9d4c49b4b8dba3bd83defdcc: Log is configured to *not* fsync() on all Append() calls
I20260928 12:23:53.625908 16637 tablet_bootstrap.cc:492] T 4de49945e6a14dbdb767ef864fffd1d4 P d798223e9d4c49b4b8dba3bd83defdcc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:23:53.626190 16637 tablet_bootstrap.cc:492] T 4de49945e6a14dbdb767ef864fffd1d4 P d798223e9d4c49b4b8dba3bd83defdcc: Bootstrap complete.
I20260928 12:23:53.627303 16637 ts_tablet_manager.cc:1397] T 4de49945e6a14dbdb767ef864fffd1d4 P d798223e9d4c49b4b8dba3bd83defdcc: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.010s
I20260928 12:23:53.628337 16637 raft_consensus.cc:348] T 4de49945e6a14dbdb767ef864fffd1d4 P d798223e9d4c49b4b8dba3bd83defdcc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42581 } } peers { permanent_uuid: "61304af90f2f400092fda93d54f75dc9" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34035 } } peers { permanent_uuid: "979f8823ef2040a283f82ad07e1ed1ca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45021 } }
I20260928 12:23:53.628561 16637 raft_consensus.cc:729] T 4de49945e6a14dbdb767ef864fffd1d4 P d798223e9d4c49b4b8dba3bd83defdcc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d798223e9d4c49b4b8dba3bd83defdcc, State: Initialized, Role: FOLLOWER
I20260928 12:23:53.628734 16637 consensus_queue.cc:260] T 4de49945e6a14dbdb767ef864fffd1d4 P d798223e9d4c49b4b8dba3bd83defdcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42581 } } peers { permanent_uuid: "61304af90f2f400092fda93d54f75dc9" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34035 } } peers { permanent_uuid: "979f8823ef2040a283f82ad07e1ed1ca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45021 } }
I20260928 12:23:53.629077 16637 ts_tablet_manager.cc:1428] T 4de49945e6a14dbdb767ef864fffd1d4 P d798223e9d4c49b4b8dba3bd83defdcc: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
Sep 28 12:23:53 dist-test-slave-920p krb5kdc[15979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598233, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:23:53.636179 16747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:53.621751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.1:36329)
0928 12:23:53.621924 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:53.621971 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:53.624460 (+ 2489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:53.625085 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:53.625090 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:53.625497 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:23:53.625763 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:53.625769 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.627409 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.627410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:53.628667 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:53.628670 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.629659 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.629661 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:53.629799 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:53.630426 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:23:53.630436 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:23:53.632143 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:23:53.634441 (+ 2298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:53.634448 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:53.634450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:53.634728 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:53.634994 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:53.634994 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:53.634995 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:53.635107 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:53.635546 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:23:53.635552 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:23:53.635697 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 12:23:53.635795 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 12:23:53.635963 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":91,"threads_started":1}
I20260928 12:23:53.636765 16748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:53.621903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:36329 (local address 127.2.56.62:39297)
0928 12:23:53.622093 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:53.622103 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:53.622151 (+ 48us) server_negotiation.cc:407] Connection header received
0928 12:23:53.624742 (+ 2591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:53.624749 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:53.624810 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:53.624942 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:23:53.625892 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.627288 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.628771 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.629457 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.629561 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:53.632266 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:23:53.632271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:23:53.632272 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:23:53.632284 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:23:53.634308 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:53.634811 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:53.634814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:53.634815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:53.634927 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:53.635167 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:53.635167 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:53.635168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:53.635450 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:23:53.635520 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:53.636357 (+ 837us) server_negotiation.cc:299] Negotiation successful
0928 12:23:53.636501 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":79,"threads_started":1}
I20260928 12:23:53.637413 16745 heartbeater.cc:347] Connected to a master server at 127.2.56.62:39297
I20260928 12:23:53.637514 16745 heartbeater.cc:464] Registering TS with master...
I20260928 12:23:53.637704 16745 heartbeater.cc:511] Master 127.2.56.62:39297 requested a full tablet report, sending...
I20260928 12:23:53.638475 16557 ts_manager.cc:194] Registered new tserver with Master: d798223e9d4c49b4b8dba3bd83defdcc (127.2.56.1:42581)
I20260928 12:23:53.639662 16557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.1@KRBTEST.COM'} at 127.2.56.1:36329
I20260928 12:23:53.640671 16745 heartbeater.cc:503] Master 127.2.56.62:39297 was elected leader, sending a full tablet report...
I20260928 12:23:53.646451 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:53.639074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:42581 (local address 127.0.0.1:59018)
0928 12:23:53.641412 (+ 2338us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:53.641453 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:53.641519 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:53.641801 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:53.641804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:53.641807 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:23:53.641895 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:53.641896 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.643060 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.643062 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:53.645121 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:53.645125 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.646109 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.646110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:53.646211 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:53.646239 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 12:23:53.646311 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 12:23:53.646373 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2303,"thread_start_us":106,"threads_started":1}
I20260928 12:23:53.646464 16753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:53.639950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59018 (local address 127.2.56.1:42581)
0928 12:23:53.641649 (+ 1699us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:53.641653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:53.641661 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:23:53.641694 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:53.641696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:53.641711 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:53.641777 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:23:53.641966 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.642982 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.645231 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.646021 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.646099 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:53.646157 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:53.646343 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 12:23:53.646389 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1662,"thread_start_us":108,"threads_started":1}
I20260928 12:23:53.647020 16679 tablet_service.cc:1514] Processing DeleteTablet for tablet 4de49945e6a14dbdb767ef864fffd1d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:23:53 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:59018
I20260928 12:23:53.647635 16754 tablet_replica.cc:330] stopping tablet replica
I20260928 12:23:53.647740 16754 raft_consensus.cc:2229] T 4de49945e6a14dbdb767ef864fffd1d4 P d798223e9d4c49b4b8dba3bd83defdcc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:23:53.647805 16754 raft_consensus.cc:2258] T 4de49945e6a14dbdb767ef864fffd1d4 P d798223e9d4c49b4b8dba3bd83defdcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:23:53.648111 16754 ts_tablet_manager.cc:1905] T 4de49945e6a14dbdb767ef864fffd1d4 P d798223e9d4c49b4b8dba3bd83defdcc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:23:53.650396 16754 ts_tablet_manager.cc:1918] T 4de49945e6a14dbdb767ef864fffd1d4 P d798223e9d4c49b4b8dba3bd83defdcc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:23:53.650478 16754 log.cc:1196] T 4de49945e6a14dbdb767ef864fffd1d4 P d798223e9d4c49b4b8dba3bd83defdcc: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/wal/wals/4de49945e6a14dbdb767ef864fffd1d4
I20260928 12:23:53.650714 16754 ts_tablet_manager.cc:1939] T 4de49945e6a14dbdb767ef864fffd1d4 P d798223e9d4c49b4b8dba3bd83defdcc: Deleting consensus metadata
I20260928 12:23:53.651413 16544 catalog_manager.cc:4841] TS d798223e9d4c49b4b8dba3bd83defdcc (127.2.56.1:42581): tablet 4de49945e6a14dbdb767ef864fffd1d4 (table test-table [id=4ec1e236e9d74ea4937f82aacc0b3a92]) successfully deleted
W20260928 12:23:53.710361 16749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:23:53.710566 16749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:23:53.710598 16749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:23:53.711782 16749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:23:53.711831 16749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:23:53.711875 16749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.2
I20260928 12:23:53.713456 16749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:43689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.2:34035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.56.2
--webserver_port=35281
--tserver_master_addrs=127.2.56.62:39297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.2
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:23:53.713689 16749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:23:53.713903 16749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:23:53.718690 16760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:23:53.718729 16762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:23:53.718878 16759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:23:53.718873 16749 server_base.cc:1034] running on GCE node
I20260928 12:23:53.719105 16749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:23:53.719339 16749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:23:53.720494 16749 hybrid_clock.cc:648] HybridClock initialized: now 1790598233720482 us; error 26 us; skew 500 ppm
Sep 28 12:23:53 dist-test-slave-920p krb5kdc[15979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598233, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:23:53.722919 16749 init.cc:377] Logged in from keytab as kudu/127.2.56.2@KRBTEST.COM (short username kudu)
I20260928 12:23:53.723371 16749 webserver.cc:460] Webserver started at http://127.2.56.2:35281/ using document root <none> and password file <none>
I20260928 12:23:53.723568 16749 fs_manager.cc:359] Metadata directory not provided
I20260928 12:23:53.723629 16749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:23:53.724951 16749 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:23:53.725672 16769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:53.725821 16749 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:53.725906 16749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/wal
uuid: "61304af90f2f400092fda93d54f75dc9"
format_stamp: "Formatted at 2026-09-28 12:23:53 on dist-test-slave-920p"
server_key: "bc731b2ecaa31fc5eccee0991f2645bc"
server_key_iv: "d94565324ec4f358b764ad68a6e25fdc"
server_key_version: "encryptionkey@0"
I20260928 12:23:53.726251 16749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:23:53.758126 16749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:23:53.760236 16749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:23:53.760376 16749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:23:53.760586 16749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:23:53.760995 16776 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 12:23:53.762490 16749 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 12:23:53.762560 16749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 12:23:53.762598 16749 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 12:23:53.763294 16749 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 12:23:53.763327 16749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 12:23:53.763432 16776 tablet_bootstrap.cc:492] T 4de49945e6a14dbdb767ef864fffd1d4 P 61304af90f2f400092fda93d54f75dc9: Bootstrap starting.
I20260928 12:23:53.764923 16776 log.cc:824] T 4de49945e6a14dbdb767ef864fffd1d4 P 61304af90f2f400092fda93d54f75dc9: Log is configured to *not* fsync() on all Append() calls
I20260928 12:23:53.768087 16776 tablet_bootstrap.cc:492] T 4de49945e6a14dbdb767ef864fffd1d4 P 61304af90f2f400092fda93d54f75dc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:23:53.768373 16776 tablet_bootstrap.cc:492] T 4de49945e6a14dbdb767ef864fffd1d4 P 61304af90f2f400092fda93d54f75dc9: Bootstrap complete.
I20260928 12:23:53.768537 16776 ts_tablet_manager.cc:1397] T 4de49945e6a14dbdb767ef864fffd1d4 P 61304af90f2f400092fda93d54f75dc9: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 12:23:53.769936 16749 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.2:34035
I20260928 12:23:53.769970 16884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.2:34035 every 8 connection(s)
I20260928 12:23:53.770320 16749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
I20260928 12:23:53.770422 16776 raft_consensus.cc:348] T 4de49945e6a14dbdb767ef864fffd1d4 P 61304af90f2f400092fda93d54f75dc9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42581 } } peers { permanent_uuid: "61304af90f2f400092fda93d54f75dc9" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34035 } } peers { permanent_uuid: "979f8823ef2040a283f82ad07e1ed1ca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45021 } }
I20260928 12:23:53.770819 16776 raft_consensus.cc:729] T 4de49945e6a14dbdb767ef864fffd1d4 P 61304af90f2f400092fda93d54f75dc9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61304af90f2f400092fda93d54f75dc9, State: Initialized, Role: FOLLOWER
I20260928 12:23:53.770939 16776 consensus_queue.cc:260] T 4de49945e6a14dbdb767ef864fffd1d4 P 61304af90f2f400092fda93d54f75dc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42581 } } peers { permanent_uuid: "61304af90f2f400092fda93d54f75dc9" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34035 } } peers { permanent_uuid: "979f8823ef2040a283f82ad07e1ed1ca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45021 } }
I20260928 12:23:53.771184 16776 ts_tablet_manager.cc:1428] T 4de49945e6a14dbdb767ef864fffd1d4 P 61304af90f2f400092fda93d54f75dc9: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
Sep 28 12:23:53 dist-test-slave-920p krb5kdc[15979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598233, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:23:53.778685 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 16749
I20260928 12:23:53.778754 16748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:53.771020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:51945 (local address 127.2.56.62:39297)
0928 12:23:53.771198 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:53.771200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:53.771389 (+ 189us) server_negotiation.cc:407] Connection header received
0928 12:23:53.771526 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:53.771527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:53.771542 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:53.771593 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:23:53.772189 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.772675 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.773664 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.773866 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.773920 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:53.776295 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:23:53.776297 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:23:53.776297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:23:53.776305 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:23:53.777558 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:53.777918 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:53.777919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:53.777919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:53.777964 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:53.778153 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:53.778154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:53.778154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:53.778246 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:23:53.778289 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:53.778573 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 12:23:53.778659 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260928 12:23:53.778867 16888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:53.771004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.2:51945)
0928 12:23:53.771235 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:53.771274 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:53.771454 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:53.771655 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:53.771659 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:53.771920 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:23:53.772100 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:53.772104 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.772770 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.772771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:53.773592 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:53.773594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.773929 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.773931 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:53.774043 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:53.774476 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:23:53.774483 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:23:53.776209 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:23:53.777625 (+ 1416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:53.777632 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:53.777633 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:53.777853 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:53.778018 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:53.778018 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:53.778018 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:53.778110 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:53.778317 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:23:53.778328 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:23:53.778453 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 12:23:53.778541 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 12:23:53.778672 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":92,"threads_started":1}
I20260928 12:23:53.778940 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.3:45021
--local_ip_for_outbound_sockets=127.2.56.3
--tserver_master_addrs=127.2.56.62:39297
--webserver_port=36961
--webserver_interface=127.2.56.3
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:43689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:23:53.779342 16885 heartbeater.cc:347] Connected to a master server at 127.2.56.62:39297
I20260928 12:23:53.779439 16885 heartbeater.cc:464] Registering TS with master...
I20260928 12:23:53.779601 16885 heartbeater.cc:511] Master 127.2.56.62:39297 requested a full tablet report, sending...
I20260928 12:23:53.780123 16557 ts_manager.cc:194] Registered new tserver with Master: 61304af90f2f400092fda93d54f75dc9 (127.2.56.2:34035)
I20260928 12:23:53.780640 16557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.2@KRBTEST.COM'} at 127.2.56.2:51945
I20260928 12:23:53.781718 16885 heartbeater.cc:503] Master 127.2.56.62:39297 was elected leader, sending a full tablet report...
Sep 28 12:23:53 dist-test-slave-920p krb5kdc[15979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598233, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for kudu/127.2.56.2@KRBTEST.COM
I20260928 12:23:53.788266 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:53.780454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:34035 (local address 127.0.0.1:37750)
0928 12:23:53.780510 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:53.780519 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:53.780552 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:53.780930 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:53.780931 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:53.781188 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:23:53.781222 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:53.781224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.782409 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.782410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:53.782927 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:53.782929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.783519 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.783522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:53.783561 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:53.784006 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:23:53.784010 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:23:53.785473 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:23:53.787298 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:53.787304 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:53.787306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:53.787483 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:53.787680 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:53.787681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:53.787681 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:53.787728 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:53.788029 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:23:53.788033 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:23:53.788075 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 12:23:53.788122 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 12:23:53.788197 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 12:23:53.788456 16890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:53.780623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37750 (local address 127.2.56.2:34035)
0928 12:23:53.780780 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:53.780784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:53.780793 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:23:53.780838 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:53.780840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:53.780853 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:53.780919 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:23:53.781313 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.782306 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.782998 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.783450 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.783524 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:53.785549 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:23:53.785558 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:23:53.785558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:23:53.785565 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:23:53.787222 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:53.787556 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:53.787559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:53.787559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:53.787622 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:53.787791 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:53.787792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:53.787792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:53.787972 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:23:53.788057 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:53.788158 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 12:23:53.788247 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260928 12:23:53.788851 16819 tablet_service.cc:1514] Processing DeleteTablet for tablet 4de49945e6a14dbdb767ef864fffd1d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:23:53 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:37750
I20260928 12:23:53.789510 16891 tablet_replica.cc:330] stopping tablet replica
I20260928 12:23:53.789611 16891 raft_consensus.cc:2229] T 4de49945e6a14dbdb767ef864fffd1d4 P 61304af90f2f400092fda93d54f75dc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:23:53.789678 16891 raft_consensus.cc:2258] T 4de49945e6a14dbdb767ef864fffd1d4 P 61304af90f2f400092fda93d54f75dc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:23:53.789987 16891 ts_tablet_manager.cc:1905] T 4de49945e6a14dbdb767ef864fffd1d4 P 61304af90f2f400092fda93d54f75dc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:23:53.791834 16891 ts_tablet_manager.cc:1918] T 4de49945e6a14dbdb767ef864fffd1d4 P 61304af90f2f400092fda93d54f75dc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:23:53.791898 16891 log.cc:1196] T 4de49945e6a14dbdb767ef864fffd1d4 P 61304af90f2f400092fda93d54f75dc9: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/wal/wals/4de49945e6a14dbdb767ef864fffd1d4
I20260928 12:23:53.792096 16891 ts_tablet_manager.cc:1939] T 4de49945e6a14dbdb767ef864fffd1d4 P 61304af90f2f400092fda93d54f75dc9: Deleting consensus metadata
I20260928 12:23:53.792532 16541 catalog_manager.cc:4841] TS 61304af90f2f400092fda93d54f75dc9 (127.2.56.2:34035): tablet 4de49945e6a14dbdb767ef864fffd1d4 (table test-table [id=4ec1e236e9d74ea4937f82aacc0b3a92]) successfully deleted
W20260928 12:23:53.860698 16889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:23:53.860910 16889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:23:53.860934 16889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:23:53.862076 16889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:23:53.862111 16889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:23:53.862136 16889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.3
I20260928 12:23:53.863531 16889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:43689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.3:45021
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.56.3
--webserver_port=36961
--tserver_master_addrs=127.2.56.62:39297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.3
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:23:53.863739 16889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:23:53.863945 16889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:23:53.868836 16896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:23:53.868891 16889 server_base.cc:1034] running on GCE node
W20260928 12:23:53.868806 16897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:23:53.868806 16899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:23:53.869201 16889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:23:53.869501 16889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:23:53.870649 16889 hybrid_clock.cc:648] HybridClock initialized: now 1790598233870641 us; error 32 us; skew 500 ppm
Sep 28 12:23:53 dist-test-slave-920p krb5kdc[15979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598233, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:23:53.872910 16889 init.cc:377] Logged in from keytab as kudu/127.2.56.3@KRBTEST.COM (short username kudu)
I20260928 12:23:53.873333 16889 webserver.cc:460] Webserver started at http://127.2.56.3:36961/ using document root <none> and password file <none>
I20260928 12:23:53.873523 16889 fs_manager.cc:359] Metadata directory not provided
I20260928 12:23:53.873598 16889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:23:53.874897 16889 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:23:53.875468 16906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:53.875607 16889 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:23:53.875685 16889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/wal
uuid: "979f8823ef2040a283f82ad07e1ed1ca"
format_stamp: "Formatted at 2026-09-28 12:23:53 on dist-test-slave-920p"
server_key: "d5a5dbf7f44c216c7970fc85674cdc47"
server_key_iv: "16dd5509fdf7c5e4c84d137b2fd3a782"
server_key_version: "encryptionkey@0"
I20260928 12:23:53.876035 16889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:23:53.905715 16889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:23:53.907787 16889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:23:53.907927 16889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:23:53.908141 16889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:23:53.908443 16889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:23:53.908476 16889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:53.908505 16889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:23:53.908527 16889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:53.913861 16889 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.3:45021
I20260928 12:23:53.913937 17019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.3:45021 every 8 connection(s)
I20260928 12:23:53.914220 16889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
Sep 28 12:23:53 dist-test-slave-920p krb5kdc[15979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598233, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:23:53.922508 16748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:53.914858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:55845 (local address 127.2.56.62:39297)
0928 12:23:53.914921 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:53.914923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:53.915127 (+ 204us) server_negotiation.cc:407] Connection header received
0928 12:23:53.915611 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:53.915611 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:53.915622 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:53.915671 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:23:53.916525 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.917008 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.917969 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.918153 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.918201 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:53.920152 (+ 1951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:23:53.920153 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:23:53.920154 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:23:53.920161 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:23:53.921376 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:53.921742 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:53.921743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:53.921743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:53.921790 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:53.921970 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:53.921971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:53.921971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:53.922071 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:23:53.922115 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:53.922371 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 12:23:53.922439 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 12:23:53.922636 17022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:53.914858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.3:55845)
0928 12:23:53.915016 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:53.915051 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:53.915530 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:53.915722 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:53.915725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:53.916264 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:23:53.916456 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:53.916461 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.917069 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.917069 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:53.917901 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:53.917905 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.918216 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.918217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:53.918320 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:53.918776 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:23:53.918784 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:23:53.920038 (+ 1254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:23:53.921463 (+ 1425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:53.921470 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:53.921472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:53.921681 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:53.921835 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:53.921836 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:53.921836 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:53.921927 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:53.922114 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:23:53.922117 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:23:53.922231 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 12:23:53.922309 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 12:23:53.922429 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":85,"threads_started":1}
I20260928 12:23:53.923117 17020 heartbeater.cc:347] Connected to a master server at 127.2.56.62:39297
I20260928 12:23:53.923204 17020 heartbeater.cc:464] Registering TS with master...
I20260928 12:23:53.923375 17020 heartbeater.cc:511] Master 127.2.56.62:39297 requested a full tablet report, sending...
I20260928 12:23:53.923735 16557 ts_manager.cc:194] Registered new tserver with Master: 979f8823ef2040a283f82ad07e1ed1ca (127.2.56.3:45021)
I20260928 12:23:53.924072 16557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.3@KRBTEST.COM'} at 127.2.56.3:55845
I20260928 12:23:53.924293 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 16889
I20260928 12:23:53.929773 16748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:53.924621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48756 (local address 127.2.56.62:39297)
0928 12:23:53.924668 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:53.924670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:53.924788 (+ 118us) server_negotiation.cc:407] Connection header received
0928 12:23:53.924861 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:53.924862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:53.924869 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:53.924912 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:23:53.925429 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.925888 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.926547 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.926721 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.926770 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:53.927751 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:23:53.927752 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:23:53.927752 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:23:53.927757 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:23:53.928851 (+ 1094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:53.929125 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:53.929126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:53.929126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:53.929169 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:53.929407 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:53.929408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:53.929408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:53.929497 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:23:53.929621 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:53.929654 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 12:23:53.929709 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 12:23:53.930606 2272 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:23:53.933131 16748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:53.930885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48772 (local address 127.2.56.62:39297)
0928 12:23:53.930920 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:53.930922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:53.931060 (+ 138us) server_negotiation.cc:407] Connection header received
0928 12:23:53.931117 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:53.931118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:53.931154 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:53.931209 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:23:53.931324 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.931769 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.932493 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.932671 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.932722 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:53.932852 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:23:53.932944 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:23:53.933011 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:53.933046 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 12:23:53.933063 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 12:23:53.933394 16557 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:48772:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:23:53.934942 16557 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:23:53.936986 16819 tablet_service.cc:1467] Processing CreateTablet for tablet f898614ae6ca4a01bb7498931e120f57 (DEFAULT_TABLE table=test-table [id=78c635ef3eef4f5781e5471f08e45de4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:23:53.937191 16819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f898614ae6ca4a01bb7498931e120f57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:23:53.938602 16776 tablet_bootstrap.cc:492] T f898614ae6ca4a01bb7498931e120f57 P 61304af90f2f400092fda93d54f75dc9: Bootstrap starting.
I20260928 12:23:53.938834 16679 tablet_service.cc:1467] Processing CreateTablet for tablet f898614ae6ca4a01bb7498931e120f57 (DEFAULT_TABLE table=test-table [id=78c635ef3eef4f5781e5471f08e45de4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:23:53.938979 16679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f898614ae6ca4a01bb7498931e120f57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:23:53.939008 16776 tablet_bootstrap.cc:654] T f898614ae6ca4a01bb7498931e120f57 P 61304af90f2f400092fda93d54f75dc9: Neither blocks nor log segments found. Creating new log.
I20260928 12:23:53.939632 16776 tablet_bootstrap.cc:492] T f898614ae6ca4a01bb7498931e120f57 P 61304af90f2f400092fda93d54f75dc9: No bootstrap required, opened a new log
I20260928 12:23:53.939682 16776 ts_tablet_manager.cc:1397] T f898614ae6ca4a01bb7498931e120f57 P 61304af90f2f400092fda93d54f75dc9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:23:53.939826 16776 raft_consensus.cc:348] T f898614ae6ca4a01bb7498931e120f57 P 61304af90f2f400092fda93d54f75dc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979f8823ef2040a283f82ad07e1ed1ca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45021 } } peers { permanent_uuid: "61304af90f2f400092fda93d54f75dc9" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34035 } } peers { permanent_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42581 } }
I20260928 12:23:53.939895 16776 raft_consensus.cc:374] T f898614ae6ca4a01bb7498931e120f57 P 61304af90f2f400092fda93d54f75dc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:23:53.939937 16776 raft_consensus.cc:729] T f898614ae6ca4a01bb7498931e120f57 P 61304af90f2f400092fda93d54f75dc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61304af90f2f400092fda93d54f75dc9, State: Initialized, Role: FOLLOWER
I20260928 12:23:53.939992 16776 consensus_queue.cc:260] T f898614ae6ca4a01bb7498931e120f57 P 61304af90f2f400092fda93d54f75dc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979f8823ef2040a283f82ad07e1ed1ca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45021 } } peers { permanent_uuid: "61304af90f2f400092fda93d54f75dc9" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34035 } } peers { permanent_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42581 } }
I20260928 12:23:53.940078 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:53.936742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:45021 (local address 127.0.0.1:52934)
0928 12:23:53.936785 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:53.936795 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:53.936838 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:53.937181 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:53.937183 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:53.937185 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:23:53.937230 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:53.937232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.938266 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.938267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:53.939067 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:53.939069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.939859 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.939860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:53.939923 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:53.939926 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:23:53.939969 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 12:23:53.940010 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 12:23:53.940088 16776 ts_tablet_manager.cc:1428] T f898614ae6ca4a01bb7498931e120f57 P 61304af90f2f400092fda93d54f75dc9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:53.940294 17025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:53.936845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52934 (local address 127.2.56.3:45021)
0928 12:23:53.937035 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:53.937039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:53.937050 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:23:53.937086 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:53.937088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:53.937103 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:53.937185 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:23:53.937356 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.938182 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.939143 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:53.939784 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:53.939853 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:53.939913 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:53.940025 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 12:23:53.940218 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":97,"threads_started":1}
I20260928 12:23:53.940513 16637 tablet_bootstrap.cc:492] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc: Bootstrap starting.
I20260928 12:23:53.940814 16637 tablet_bootstrap.cc:654] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc: Neither blocks nor log segments found. Creating new log.
I20260928 12:23:53.941176 16954 tablet_service.cc:1467] Processing CreateTablet for tablet f898614ae6ca4a01bb7498931e120f57 (DEFAULT_TABLE table=test-table [id=78c635ef3eef4f5781e5471f08e45de4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:23:53.941344 16637 tablet_bootstrap.cc:492] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc: No bootstrap required, opened a new log
I20260928 12:23:53.941380 16637 ts_tablet_manager.cc:1397] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 12:23:53.941468 16637 raft_consensus.cc:348] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979f8823ef2040a283f82ad07e1ed1ca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45021 } } peers { permanent_uuid: "61304af90f2f400092fda93d54f75dc9" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34035 } } peers { permanent_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42581 } }
I20260928 12:23:53.941525 16637 raft_consensus.cc:374] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:23:53.941526 16954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f898614ae6ca4a01bb7498931e120f57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:23:53.941552 16637 raft_consensus.cc:729] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d798223e9d4c49b4b8dba3bd83defdcc, State: Initialized, Role: FOLLOWER
I20260928 12:23:53.941594 16637 consensus_queue.cc:260] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979f8823ef2040a283f82ad07e1ed1ca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45021 } } peers { permanent_uuid: "61304af90f2f400092fda93d54f75dc9" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34035 } } peers { permanent_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42581 } }
I20260928 12:23:53.941677 16637 ts_tablet_manager.cc:1428] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:53.944212 17028 tablet_bootstrap.cc:492] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca: Bootstrap starting.
I20260928 12:23:53.944720 17028 tablet_bootstrap.cc:654] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca: Neither blocks nor log segments found. Creating new log.
I20260928 12:23:53.945117 17028 log.cc:824] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca: Log is configured to *not* fsync() on all Append() calls
I20260928 12:23:53.945797 17028 tablet_bootstrap.cc:492] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca: No bootstrap required, opened a new log
I20260928 12:23:53.945860 17028 ts_tablet_manager.cc:1397] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 12:23:53.947139 17028 raft_consensus.cc:348] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979f8823ef2040a283f82ad07e1ed1ca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45021 } } peers { permanent_uuid: "61304af90f2f400092fda93d54f75dc9" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34035 } } peers { permanent_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42581 } }
I20260928 12:23:53.947268 17028 raft_consensus.cc:374] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:23:53.947297 17028 raft_consensus.cc:729] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 979f8823ef2040a283f82ad07e1ed1ca, State: Initialized, Role: FOLLOWER
I20260928 12:23:53.947412 17028 consensus_queue.cc:260] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979f8823ef2040a283f82ad07e1ed1ca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45021 } } peers { permanent_uuid: "61304af90f2f400092fda93d54f75dc9" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34035 } } peers { permanent_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42581 } }
I20260928 12:23:53.947634 17028 ts_tablet_manager.cc:1428] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 12:23:53.947680 17020 heartbeater.cc:503] Master 127.2.56.62:39297 was elected leader, sending a full tablet report...
W20260928 12:23:54.021158 16886 tablet.cc:2369] T f898614ae6ca4a01bb7498931e120f57 P 61304af90f2f400092fda93d54f75dc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:23:54.038964 17030 raft_consensus.cc:482] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:23:54.039104 17030 raft_consensus.cc:504] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979f8823ef2040a283f82ad07e1ed1ca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45021 } } peers { permanent_uuid: "61304af90f2f400092fda93d54f75dc9" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34035 } } peers { permanent_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42581 } }
I20260928 12:23:54.039561 17030 leader_election.cc:288] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61304af90f2f400092fda93d54f75dc9 (127.2.56.2:34035), d798223e9d4c49b4b8dba3bd83defdcc (127.2.56.1:42581)
I20260928 12:23:54.042356 17022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:54.039768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:42581 (local address 127.2.56.3:35085)
0928 12:23:54.039863 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:54.039876 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:54.039936 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:54.040096 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:54.040097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:54.040099 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:23:54.040163 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:54.040165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:54.040793 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:54.040794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:54.041603 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:54.041606 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:54.042117 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:54.042118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:54.042189 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:54.042192 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:23:54.042237 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 12:23:54.042276 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":22}
I20260928 12:23:54.042362 16753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:54.039834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:35085 (local address 127.2.56.1:42581)
0928 12:23:54.039876 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:54.039878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:54.039925 (+ 47us) server_negotiation.cc:407] Connection header received
0928 12:23:54.040005 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:54.040006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:54.040019 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:54.040079 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:23:54.040226 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:54.040713 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:54.041688 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:54.042044 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:54.042104 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:54.042116 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:54.042260 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 12:23:54.042303 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:23:54.042588 17031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:54.039799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:34035 (local address 127.2.56.3:53787)
0928 12:23:54.040004 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:54.040013 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:54.040064 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:54.040210 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:54.040211 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:54.040213 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:23:54.040281 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:54.040283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:54.040970 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:54.040971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:54.041776 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:54.041778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:54.042349 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:54.042350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:54.042415 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:54.042418 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:23:54.042468 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 12:23:54.042504 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":2,"thread_start_us":145,"threads_started":1}
I20260928 12:23:54.042585 16890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:54.039882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:53787 (local address 127.2.56.2:34035)
0928 12:23:54.039930 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:54.039932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:54.040063 (+ 131us) server_negotiation.cc:407] Connection header received
0928 12:23:54.040112 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:54.040114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:54.040138 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:54.040195 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:23:54.040367 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:54.040891 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:54.041861 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:54.042268 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:54.042327 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:54.042418 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:54.042471 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 12:23:54.042510 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:23:54.042822 16699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f898614ae6ca4a01bb7498931e120f57" candidate_uuid: "979f8823ef2040a283f82ad07e1ed1ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" is_pre_election: true
I20260928 12:23:54.043000 16699 raft_consensus.cc:2454] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 979f8823ef2040a283f82ad07e1ed1ca in term 0.
I20260928 12:23:54.043020 16839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f898614ae6ca4a01bb7498931e120f57" candidate_uuid: "979f8823ef2040a283f82ad07e1ed1ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61304af90f2f400092fda93d54f75dc9" is_pre_election: true
I20260928 12:23:54.043155 16839 raft_consensus.cc:2454] T f898614ae6ca4a01bb7498931e120f57 P 61304af90f2f400092fda93d54f75dc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 979f8823ef2040a283f82ad07e1ed1ca in term 0.
I20260928 12:23:54.043257 16910 leader_election.cc:302] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 979f8823ef2040a283f82ad07e1ed1ca, d798223e9d4c49b4b8dba3bd83defdcc; no voters:
I20260928 12:23:54.043380 17030 raft_consensus.cc:2781] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:23:54.043437 17030 raft_consensus.cc:482] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:23:54.043452 17030 raft_consensus.cc:3037] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:23:54.044152 17030 raft_consensus.cc:504] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979f8823ef2040a283f82ad07e1ed1ca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45021 } } peers { permanent_uuid: "61304af90f2f400092fda93d54f75dc9" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34035 } } peers { permanent_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42581 } }
I20260928 12:23:54.044272 17030 leader_election.cc:288] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca [CANDIDATE]: Term 1 election: Requested vote from peers 61304af90f2f400092fda93d54f75dc9 (127.2.56.2:34035), d798223e9d4c49b4b8dba3bd83defdcc (127.2.56.1:42581)
I20260928 12:23:54.044425 16839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f898614ae6ca4a01bb7498931e120f57" candidate_uuid: "979f8823ef2040a283f82ad07e1ed1ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61304af90f2f400092fda93d54f75dc9"
I20260928 12:23:54.044442 16699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f898614ae6ca4a01bb7498931e120f57" candidate_uuid: "979f8823ef2040a283f82ad07e1ed1ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d798223e9d4c49b4b8dba3bd83defdcc"
I20260928 12:23:54.044508 16839 raft_consensus.cc:3037] T f898614ae6ca4a01bb7498931e120f57 P 61304af90f2f400092fda93d54f75dc9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:23:54.044509 16699 raft_consensus.cc:3037] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:23:54.045336 16839 raft_consensus.cc:2454] T f898614ae6ca4a01bb7498931e120f57 P 61304af90f2f400092fda93d54f75dc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 979f8823ef2040a283f82ad07e1ed1ca in term 1.
I20260928 12:23:54.045507 16907 leader_election.cc:302] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61304af90f2f400092fda93d54f75dc9, 979f8823ef2040a283f82ad07e1ed1ca; no voters:
I20260928 12:23:54.045604 17030 raft_consensus.cc:2781] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:23:54.045691 16699 raft_consensus.cc:2454] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 979f8823ef2040a283f82ad07e1ed1ca in term 1.
I20260928 12:23:54.045759 17030 raft_consensus.cc:686] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca [term 1 LEADER]: Becoming Leader. State: Replica: 979f8823ef2040a283f82ad07e1ed1ca, State: Running, Role: LEADER
I20260928 12:23:54.045851 17030 consensus_queue.cc:237] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979f8823ef2040a283f82ad07e1ed1ca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45021 } } peers { permanent_uuid: "61304af90f2f400092fda93d54f75dc9" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34035 } } peers { permanent_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42581 } }
I20260928 12:23:54.046693 16556 catalog_manager.cc:5495] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 979f8823ef2040a283f82ad07e1ed1ca (127.2.56.3). New cstate: current_term: 1 leader_uuid: "979f8823ef2040a283f82ad07e1ed1ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979f8823ef2040a283f82ad07e1ed1ca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61304af90f2f400092fda93d54f75dc9" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42581 } health_report { overall_health: UNKNOWN } } }
I20260928 12:23:54.056360 17025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:54.054194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52942 (local address 127.2.56.3:45021)
0928 12:23:54.054236 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:54.054238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:54.054247 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:23:54.054308 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:54.054309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:54.054327 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:54.054381 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:23:54.054491 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:54.054955 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:54.055654 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:54.055843 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:54.055895 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:54.056035 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:23:54.056131 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:23:54.056202 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:54.056239 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 12:23:54.056262 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 12:23:54.058130 16839 raft_consensus.cc:1261] T f898614ae6ca4a01bb7498931e120f57 P 61304af90f2f400092fda93d54f75dc9 [term 1 FOLLOWER]: Refusing update from remote peer 979f8823ef2040a283f82ad07e1ed1ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:23:54.058141 16699 raft_consensus.cc:1261] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc [term 1 FOLLOWER]: Refusing update from remote peer 979f8823ef2040a283f82ad07e1ed1ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:23:54.058480 17032 consensus_queue.cc:1035] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "61304af90f2f400092fda93d54f75dc9" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:23:54.058602 17030 consensus_queue.cc:1035] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:23:54.060599 17034 mvcc.cc:205] Tried to move back new op lower bound from 7334290366699630592 to 7334290366652514304. Current Snapshot: MvccSnapshot[applied={T|T < 7334290366699630592}]
I20260928 12:23:54.060851 17035 mvcc.cc:205] Tried to move back new op lower bound from 7334290366699630592 to 7334290366652514304. Current Snapshot: MvccSnapshot[applied={T|T < 7334290366699630592}]
I20260928 12:23:54.063462 16557 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:48772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:23:54.063580 16557 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:48772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:23:54.064491 16557 catalog_manager.cc:5782] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757: Sending DeleteTablet for 3 replicas of tablet f898614ae6ca4a01bb7498931e120f57
I20260928 12:23:54.064837 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 16595
I20260928 12:23:54.064836 16819 tablet_service.cc:1514] Processing DeleteTablet for tablet f898614ae6ca4a01bb7498931e120f57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:23:54 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:37750
I20260928 12:23:54.064826 16679 tablet_service.cc:1514] Processing DeleteTablet for tablet f898614ae6ca4a01bb7498931e120f57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:23:54 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:59018
I20260928 12:23:54.064932 16754 tablet_replica.cc:330] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc: stopping tablet replica
I20260928 12:23:54.064976 16754 raft_consensus.cc:2229] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:23:54.065026 16754 raft_consensus.cc:2258] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:23:54.065296 16891 tablet_replica.cc:330] T f898614ae6ca4a01bb7498931e120f57 P 61304af90f2f400092fda93d54f75dc9: stopping tablet replica
I20260928 12:23:54.065366 16891 raft_consensus.cc:2229] T f898614ae6ca4a01bb7498931e120f57 P 61304af90f2f400092fda93d54f75dc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:23:54.065407 16891 raft_consensus.cc:2258] T f898614ae6ca4a01bb7498931e120f57 P 61304af90f2f400092fda93d54f75dc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:23:54.065559 16954 tablet_service.cc:1514] Processing DeleteTablet for tablet f898614ae6ca4a01bb7498931e120f57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:23:54 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:52934
I20260928 12:23:54.065773 16891 ts_tablet_manager.cc:1905] T f898614ae6ca4a01bb7498931e120f57 P 61304af90f2f400092fda93d54f75dc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:23:54.067862 16891 ts_tablet_manager.cc:1918] T f898614ae6ca4a01bb7498931e120f57 P 61304af90f2f400092fda93d54f75dc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:23:54.067920 16891 log.cc:1196] T f898614ae6ca4a01bb7498931e120f57 P 61304af90f2f400092fda93d54f75dc9: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/wal/wals/f898614ae6ca4a01bb7498931e120f57
I20260928 12:23:54.068099 16891 ts_tablet_manager.cc:1939] T f898614ae6ca4a01bb7498931e120f57 P 61304af90f2f400092fda93d54f75dc9: Deleting consensus metadata
I20260928 12:23:54.068393 16541 catalog_manager.cc:4841] TS 61304af90f2f400092fda93d54f75dc9 (127.2.56.2:34035): tablet f898614ae6ca4a01bb7498931e120f57 (table test-table [id=78c635ef3eef4f5781e5471f08e45de4]) successfully deleted
I20260928 12:23:54.071036 17044 tablet_replica.cc:330] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca: stopping tablet replica
W20260928 12:23:54.071022 16544 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.56.1:42581 (error 108)
I20260928 12:23:54.071079 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 16749
I20260928 12:23:54.071161 17044 raft_consensus.cc:2229] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:23:54.072741 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:54.072606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:42581 (local address 127.0.0.1:59032)
0928 12:23:54.072671 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:54.072708 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.1:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 12:23:54.072932 16544 catalog_manager.cc:4584] TS d798223e9d4c49b4b8dba3bd83defdcc (127.2.56.1:42581): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f898614ae6ca4a01bb7498931e120f57: Network error: Client connection negotiation failed: client connection to 127.2.56.1:42581: connect: Connection refused (error 111)
I20260928 12:23:54.075258 17044 raft_consensus.cc:2258] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:23:54.075531 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 16889
I20260928 12:23:54.080193 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 16524
I20260928 12:23:54.080346 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:54.080180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:45021 (local address 127.0.0.1:52946)
0928 12:23:54.080262 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:54.080304 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.3:45021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 12:23:54.085039 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:39297
--webserver_interface=127.2.56.62
--webserver_port=35425
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:43689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:39297
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:23:54.164759 17046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:23:54.164975 17046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:23:54.164996 17046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:23:54.166262 17046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:23:54.166307 17046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:23:54.166320 17046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:23:54.166332 17046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:23:54.166342 17046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:23:54.167846 17046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:43689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:39297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:39297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=35425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:23:54.168063 17046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:23:54.168253 17046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:23:54.173055 17054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:23:54.173183 17051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:23:54.173085 17046 server_base.cc:1034] running on GCE node
W20260928 12:23:54.173295 17052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:23:54.173719 17046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:23:54.174028 17046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:23:54.175192 17046 hybrid_clock.cc:648] HybridClock initialized: now 1790598234175177 us; error 28 us; skew 500 ppm
Sep 28 12:23:54 dist-test-slave-920p krb5kdc[15979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598234, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:23:54.177600 17046 init.cc:377] Logged in from keytab as kudu/127.2.56.62@KRBTEST.COM (short username kudu)
I20260928 12:23:54.178097 17046 webserver.cc:460] Webserver started at http://127.2.56.62:35425/ using document root <none> and password file <none>
I20260928 12:23:54.178292 17046 fs_manager.cc:359] Metadata directory not provided
I20260928 12:23:54.178335 17046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:23:54.179605 17046 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:23:54.180181 17061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:54.180330 17046 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:54.180406 17046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "edfd1d9e4b2845bd80adc01266c03757"
format_stamp: "Formatted at 2026-09-28 12:23:52 on dist-test-slave-920p"
server_key: "4180f1c2b871f2b365549cdd892b07f8"
server_key_iv: "57d546bbe2952e0e637b69c48fe6e1bf"
server_key_version: "encryptionkey@0"
I20260928 12:23:54.180775 17046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:23:54.201792 17046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:23:54.203616 17046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:23:54.203749 17046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:23:54.207624 17046 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:39297
I20260928 12:23:54.207661 17113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:39297 every 8 connection(s)
I20260928 12:23:54.208002 17046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:23:54.209604 17114 sys_catalog.cc:263] Verifying existing consensus state
I20260928 12:23:54.209678 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 17046
I20260928 12:23:54.209955 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.1:42581
--local_ip_for_outbound_sockets=127.2.56.1
--tserver_master_addrs=127.2.56.62:39297
--webserver_port=39933
--webserver_interface=127.2.56.1
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:43689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:23:54.210485 17114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757: Bootstrap starting.
I20260928 12:23:54.218668 17114 log.cc:824] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757: Log is configured to *not* fsync() on all Append() calls
I20260928 12:23:54.220789 17114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:23:54.221061 17114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757: Bootstrap complete.
I20260928 12:23:54.223246 17114 raft_consensus.cc:348] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edfd1d9e4b2845bd80adc01266c03757" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39297 } }
I20260928 12:23:54.223479 17114 raft_consensus.cc:729] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: edfd1d9e4b2845bd80adc01266c03757, State: Initialized, Role: FOLLOWER
I20260928 12:23:54.223625 17114 consensus_queue.cc:260] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edfd1d9e4b2845bd80adc01266c03757" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39297 } }
I20260928 12:23:54.223701 17114 raft_consensus.cc:388] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:23:54.223749 17114 raft_consensus.cc:482] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:23:54.223796 17114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [term 2 FOLLOWER]: Advancing to term 3
I20260928 12:23:54.224746 17114 raft_consensus.cc:504] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edfd1d9e4b2845bd80adc01266c03757" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39297 } }
I20260928 12:23:54.224841 17114 leader_election.cc:302] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edfd1d9e4b2845bd80adc01266c03757; no voters:
I20260928 12:23:54.225008 17114 leader_election.cc:288] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 12:23:54.225072 17118 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 12:23:54.225278 17114 sys_catalog.cc:564] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:23:54.225312 17118 raft_consensus.cc:686] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [term 3 LEADER]: Becoming Leader. State: Replica: edfd1d9e4b2845bd80adc01266c03757, State: Running, Role: LEADER
I20260928 12:23:54.225422 17118 consensus_queue.cc:237] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edfd1d9e4b2845bd80adc01266c03757" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39297 } }
I20260928 12:23:54.225728 17118 sys_catalog.cc:455] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [sys.catalog]: SysCatalogTable state changed. Reason: New leader edfd1d9e4b2845bd80adc01266c03757. Latest consensus state: current_term: 3 leader_uuid: "edfd1d9e4b2845bd80adc01266c03757" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edfd1d9e4b2845bd80adc01266c03757" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39297 } } }
I20260928 12:23:54.225809 17118 sys_catalog.cc:458] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [sys.catalog]: This master's current role is: LEADER
I20260928 12:23:54.225802 17119 sys_catalog.cc:455] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "edfd1d9e4b2845bd80adc01266c03757" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edfd1d9e4b2845bd80adc01266c03757" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39297 } } }
I20260928 12:23:54.226079 17119 sys_catalog.cc:458] T 00000000000000000000000000000000 P edfd1d9e4b2845bd80adc01266c03757 [sys.catalog]: This master's current role is: LEADER
I20260928 12:23:54.226549 17125 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:23:54.227300 17125 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:23:54.227394 17125 catalog_manager.cc:1259] Loaded cluster ID: a95b5298100148eca2ca97821ac794e3
I20260928 12:23:54.227427 17125 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 12:23:54.227643 17125 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa2b145c9ad google::LogMessage::Fail() at ??:0
@ 0x7fa2b145f88a google::LogMessage::SendToLog() at ??:0
@ 0x7fa2b145c4de google::LogMessage::Flush() at ??:0
@ 0x7fa2b145da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa2b937d0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fa2b93868b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa2b332a0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa2b331f954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa2b41836db start_thread at ??:0
@ 0x7fa2aff5a61f clone at ??:0
W20260928 12:23:54.300279 17116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:23:54.300493 17116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:23:54.300529 17116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:23:54.301797 17116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:23:54.301851 17116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:23:54.301889 17116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.1
I20260928 12:23:54.303411 17116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:43689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.1:42581
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.56.1
--webserver_port=39933
--tserver_master_addrs=127.2.56.62:39297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.1
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:23:54.303653 17116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:23:54.303871 17116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:23:54.308827 17153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:23:54.308807 17151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:23:54.308969 17150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:23:54.309108 17116 server_base.cc:1034] running on GCE node
I20260928 12:23:54.309418 17116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:23:54.309653 17116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:23:54.310817 17116 hybrid_clock.cc:648] HybridClock initialized: now 1790598234310792 us; error 32 us; skew 500 ppm
Sep 28 12:23:54 dist-test-slave-920p krb5kdc[15979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598234, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:23:54.313333 17116 init.cc:377] Logged in from keytab as kudu/127.2.56.1@KRBTEST.COM (short username kudu)
I20260928 12:23:54.313752 17116 webserver.cc:460] Webserver started at http://127.2.56.1:39933/ using document root <none> and password file <none>
I20260928 12:23:54.313969 17116 fs_manager.cc:359] Metadata directory not provided
I20260928 12:23:54.314018 17116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:23:54.315804 17116 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:23:54.316571 17160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:54.316820 17116 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:23:54.316911 17116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/wal
uuid: "d798223e9d4c49b4b8dba3bd83defdcc"
format_stamp: "Formatted at 2026-09-28 12:23:52 on dist-test-slave-920p"
server_key: "5147b34ccacee2e75a5c1e03c3893046"
server_key_iv: "18f08ad9345d54a9d893e06ebce2539c"
server_key_version: "encryptionkey@0"
I20260928 12:23:54.317298 17116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:23:54.334954 17116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:23:54.337041 17116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:23:54.337177 17116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:23:54.337421 17116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:23:54.337817 17167 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 12:23:54.339566 17116 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 12:23:54.339614 17116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 12:23:54.339649 17116 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 12:23:54.340256 17116 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 12:23:54.340288 17116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 12:23:54.340395 17167 tablet_bootstrap.cc:492] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc: Bootstrap starting.
I20260928 12:23:54.346927 17116 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.1:42581
I20260928 12:23:54.347069 17274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.1:42581 every 8 connection(s)
I20260928 12:23:54.347344 17116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
I20260928 12:23:54.351451 17167 log.cc:824] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc: Log is configured to *not* fsync() on all Append() calls
I20260928 12:23:54.354589 17167 tablet_bootstrap.cc:492] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:23:54.354944 17167 tablet_bootstrap.cc:492] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc: Bootstrap complete.
I20260928 12:23:54.354974 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 17116
I20260928 12:23:54.355187 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.2:34035
--local_ip_for_outbound_sockets=127.2.56.2
--tserver_master_addrs=127.2.56.62:39297
--webserver_port=35281
--webserver_interface=127.2.56.2
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:43689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:23:54.356076 17167 ts_tablet_manager.cc:1397] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.009s
I20260928 12:23:54.357667 17167 raft_consensus.cc:348] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979f8823ef2040a283f82ad07e1ed1ca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45021 } } peers { permanent_uuid: "61304af90f2f400092fda93d54f75dc9" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34035 } } peers { permanent_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42581 } }
I20260928 12:23:54.358004 17167 raft_consensus.cc:729] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d798223e9d4c49b4b8dba3bd83defdcc, State: Initialized, Role: FOLLOWER
I20260928 12:23:54.358184 17167 consensus_queue.cc:260] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979f8823ef2040a283f82ad07e1ed1ca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45021 } } peers { permanent_uuid: "61304af90f2f400092fda93d54f75dc9" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34035 } } peers { permanent_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42581 } }
I20260928 12:23:54.358492 17167 ts_tablet_manager.cc:1428] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
W20260928 12:23:54.436458 17279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:23:54.436671 17279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:23:54.436704 17279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:23:54.438015 17279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:23:54.438062 17279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:23:54.438097 17279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.2
I20260928 12:23:54.440135 17279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:43689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.2:34035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.56.2
--webserver_port=35281
--tserver_master_addrs=127.2.56.62:39297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.2
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:23:54.440393 17279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:23:54.440646 17279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:23:54.442780 17277 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:23:54.347936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.1:59341)
0928 12:23:54.348095 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:54.348138 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:54.348600 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:54.442596 (+ 93996us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":70,"threads_started":1}
W20260928 12:23:54.443014 17275 heartbeater.cc:644] Failed to heartbeat to 127.2.56.62:39297 (0 consecutive failures): Network error: Failed to ping master at 127.2.56.62:39297: Client connection negotiation failed: client connection to 127.2.56.62:39297: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 12:23:54.443311 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:54.443206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.1:39613)
0928 12:23:54.443255 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:54.443290 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 12:23:54.443634 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:54.443558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.1:32839)
0928 12:23:54.443598 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:54.443618 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 12:23:54.443729 17275 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 12:23:54.445987 17288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:23:54.445971 17286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:23:54.446004 17285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:23:54.446269 17279 server_base.cc:1034] running on GCE node
I20260928 12:23:54.446437 17279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:23:54.446630 17279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:23:54.447783 17279 hybrid_clock.cc:648] HybridClock initialized: now 1790598234447766 us; error 26 us; skew 500 ppm
Sep 28 12:23:54 dist-test-slave-920p krb5kdc[15979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598234, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:23:54.450132 17279 init.cc:377] Logged in from keytab as kudu/127.2.56.2@KRBTEST.COM (short username kudu)
I20260928 12:23:54.450518 17279 webserver.cc:460] Webserver started at http://127.2.56.2:35281/ using document root <none> and password file <none>
I20260928 12:23:54.450688 17279 fs_manager.cc:359] Metadata directory not provided
I20260928 12:23:54.450726 17279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:23:54.451933 17279 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:23:54.452405 17295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:54.452545 17279 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:54.452617 17279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/wal
uuid: "61304af90f2f400092fda93d54f75dc9"
format_stamp: "Formatted at 2026-09-28 12:23:53 on dist-test-slave-920p"
server_key: "bc731b2ecaa31fc5eccee0991f2645bc"
server_key_iv: "d94565324ec4f358b764ad68a6e25fdc"
server_key_version: "encryptionkey@0"
I20260928 12:23:54.452955 17279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:23:54.462256 17279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:23:54.463910 17279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:23:54.464023 17279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:23:54.464226 17279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:23:54.464497 17279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:23:54.464527 17279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:54.464548 17279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:23:54.464571 17279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:54.470332 17279 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.2:34035
I20260928 12:23:54.470427 17408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.2:34035 every 8 connection(s)
I20260928 12:23:54.470712 17279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
I20260928 12:23:54.471709 17411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:54.471344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.2:35565)
0928 12:23:54.471493 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:54.471565 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
W20260928 12:23:54.472086 17409 heartbeater.cc:644] Failed to heartbeat to 127.2.56.62:39297 (0 consecutive failures): Network error: Failed to ping master at 127.2.56.62:39297: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
I20260928 12:23:54.472338 17411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:54.472245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.2:48877)
0928 12:23:54.472287 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:54.472315 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 12:23:54.472558 17411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:54.472501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.2:52929)
0928 12:23:54.472521 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:54.472538 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260928 12:23:54.472641 17409 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 12:23:54.480100 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 17279
I20260928 12:23:54.480317 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.3:45021
--local_ip_for_outbound_sockets=127.2.56.3
--tserver_master_addrs=127.2.56.62:39297
--webserver_port=36961
--webserver_interface=127.2.56.3
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:43689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:23:54.561008 17412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:23:54.561224 17412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:23:54.561249 17412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:23:54.562422 17412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:23:54.562461 17412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:23:54.562486 17412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.3
I20260928 12:23:54.563905 17412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:43689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.3:45021
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.56.3
--webserver_port=36961
--tserver_master_addrs=127.2.56.62:39297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.3
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:23:54.564108 17412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:23:54.564293 17412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:23:54.569190 17420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:23:54.569201 17417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:23:54.569334 17412 server_base.cc:1034] running on GCE node
W20260928 12:23:54.569212 17418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:23:54.569609 17412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:23:54.569828 17412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:23:54.570976 17412 hybrid_clock.cc:648] HybridClock initialized: now 1790598234570963 us; error 30 us; skew 500 ppm
Sep 28 12:23:54 dist-test-slave-920p krb5kdc[15979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598234, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:23:54.573297 17412 init.cc:377] Logged in from keytab as kudu/127.2.56.3@KRBTEST.COM (short username kudu)
I20260928 12:23:54.573731 17412 webserver.cc:460] Webserver started at http://127.2.56.3:36961/ using document root <none> and password file <none>
I20260928 12:23:54.573948 17412 fs_manager.cc:359] Metadata directory not provided
I20260928 12:23:54.573997 17412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:23:54.575359 17412 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:23:54.575977 17427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:23:54.576121 17412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:23:54.576201 17412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/wal
uuid: "979f8823ef2040a283f82ad07e1ed1ca"
format_stamp: "Formatted at 2026-09-28 12:23:53 on dist-test-slave-920p"
server_key: "d5a5dbf7f44c216c7970fc85674cdc47"
server_key_iv: "16dd5509fdf7c5e4c84d137b2fd3a782"
server_key_version: "encryptionkey@0"
I20260928 12:23:54.576536 17412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:23:54.594974 17412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:23:54.596777 17412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:23:54.596910 17412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:23:54.597134 17412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:23:54.597558 17434 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 12:23:54.599144 17412 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 12:23:54.599188 17412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 12:23:54.599222 17412 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 12:23:54.599822 17412 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 12:23:54.599852 17412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 12:23:54.599927 17434 tablet_bootstrap.cc:492] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca: Bootstrap starting.
I20260928 12:23:54.601405 17434 log.cc:824] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca: Log is configured to *not* fsync() on all Append() calls
I20260928 12:23:54.603791 17434 tablet_bootstrap.cc:492] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:23:54.604038 17434 tablet_bootstrap.cc:492] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca: Bootstrap complete.
I20260928 12:23:54.604173 17434 ts_tablet_manager.cc:1397] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260928 12:23:54.605600 17434 raft_consensus.cc:348] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979f8823ef2040a283f82ad07e1ed1ca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45021 } } peers { permanent_uuid: "61304af90f2f400092fda93d54f75dc9" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34035 } } peers { permanent_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42581 } }
I20260928 12:23:54.605831 17434 raft_consensus.cc:729] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 979f8823ef2040a283f82ad07e1ed1ca, State: Initialized, Role: FOLLOWER
I20260928 12:23:54.605995 17434 consensus_queue.cc:260] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979f8823ef2040a283f82ad07e1ed1ca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45021 } } peers { permanent_uuid: "61304af90f2f400092fda93d54f75dc9" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34035 } } peers { permanent_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42581 } }
I20260928 12:23:54.606282 17412 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.3:45021
I20260928 12:23:54.606335 17434 ts_tablet_manager.cc:1428] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
I20260928 12:23:54.606500 17543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.3:45021 every 8 connection(s)
I20260928 12:23:54.606612 17412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
I20260928 12:23:54.607486 17546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:54.607165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.3:60549)
0928 12:23:54.607298 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:54.607361 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":62,"threads_started":1}
W20260928 12:23:54.607676 17544 heartbeater.cc:644] Failed to heartbeat to 127.2.56.62:39297 (0 consecutive failures): Network error: Failed to ping master at 127.2.56.62:39297: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
I20260928 12:23:54.607918 17546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:54.607828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.3:48985)
0928 12:23:54.607871 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:54.607894 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 12:23:54.608170 17546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:54.608093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.3:49405)
0928 12:23:54.608128 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:54.608149 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 12:23:54.608325 17544 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 12:23:54.615293 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 17412
I20260928 12:23:55.444499 17548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:55.444209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.1:36983)
0928 12:23:55.444369 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:55.444433 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
I20260928 12:23:55.473290 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:55.473001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.2:46021)
0928 12:23:55.473155 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:55.473207 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":77,"threads_started":1}
I20260928 12:23:55.609021 17550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:55.608740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.3:42065)
0928 12:23:55.608905 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:55.608959 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":84,"threads_started":1}
I20260928 12:23:55.937587 17551 raft_consensus.cc:482] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:23:55.937743 17551 raft_consensus.cc:504] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979f8823ef2040a283f82ad07e1ed1ca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45021 } } peers { permanent_uuid: "61304af90f2f400092fda93d54f75dc9" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34035 } } peers { permanent_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42581 } }
I20260928 12:23:55.938088 17551 leader_election.cc:288] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 979f8823ef2040a283f82ad07e1ed1ca (127.2.56.3:45021), 61304af90f2f400092fda93d54f75dc9 (127.2.56.2:34035)
Sep 28 12:23:55 dist-test-slave-920p krb5kdc[15979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598234, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for kudu/127.2.56.2@KRBTEST.COM
Sep 28 12:23:55 dist-test-slave-920p krb5kdc[15979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598234, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for kudu/127.2.56.3@KRBTEST.COM
I20260928 12:23:55.948776 17553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:55.938467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:51393 (local address 127.2.56.2:34035)
0928 12:23:55.938694 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:55.938709 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:55.938741 (+ 32us) server_negotiation.cc:407] Connection header received
0928 12:23:55.939416 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:55.939420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:55.939478 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:55.939645 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:23:55.940098 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:55.941129 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:55.942155 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:55.942662 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:55.942767 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:55.945108 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:23:55.945114 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:23:55.945114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:23:55.945124 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:23:55.947104 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:55.947547 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:55.947550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:55.947551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:55.947658 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:55.947907 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:55.947907 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:55.947908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:55.948186 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:23:55.948256 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:55.948534 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 12:23:55.948655 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":87,"threads_started":1}
I20260928 12:23:55.948766 17548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:55.938301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:34035 (local address 127.2.56.1:51393)
0928 12:23:55.938372 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:55.938385 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:55.938451 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:55.939654 (+ 1203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:55.939656 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:55.939852 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:23:55.939925 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:55.939928 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:55.941247 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:55.941248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:55.942061 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:55.942064 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:55.942745 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:55.942745 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:55.942879 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:55.943407 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:23:55.943418 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:23:55.945000 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:23:55.947204 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:55.947212 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:55.947214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:55.947464 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:55.947732 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:55.947733 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:55.947734 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:55.947845 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:55.948252 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:23:55.948260 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:23:55.948386 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 12:23:55.948475 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 12:23:55.948657 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":5}
I20260928 12:23:55.949151 17552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:55.938367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:45021 (local address 127.2.56.1:36751)
0928 12:23:55.938550 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:55.938563 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:55.938633 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:55.939054 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:55.939058 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:55.939532 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:23:55.939845 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:55.939854 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:55.941100 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:55.941101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:55.942065 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:55.942068 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:55.942833 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:55.942834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:55.942908 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:55.943598 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:23:55.943603 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:23:55.945333 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:23:55.947618 (+ 2285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:55.947619 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:55.947619 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:55.947838 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:55.948223 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:55.948225 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:55.948225 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:55.948295 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:55.948752 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:23:55.948753 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:23:55.948806 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 12:23:55.948880 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 12:23:55.949031 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":1,"thread_start_us":66,"threads_started":1}
W20260928 12:23:55.949628 17161 leader_election.cc:341] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 61304af90f2f400092fda93d54f75dc9 (127.2.56.2:34035): Not found: Tablet not found: f898614ae6ca4a01bb7498931e120f57
I20260928 12:23:55.949734 17554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:55.938521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:36751 (local address 127.2.56.3:45021)
0928 12:23:55.938674 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:55.938684 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:55.938705 (+ 21us) server_negotiation.cc:407] Connection header received
0928 12:23:55.938819 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:55.938821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:55.938865 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:55.938980 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:23:55.939989 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:55.940986 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:55.942168 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:55.942708 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:55.942852 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:55.945472 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:23:55.945477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:23:55.945478 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:23:55.945489 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:23:55.947509 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:55.947985 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:55.947990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:55.947990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:55.948149 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:55.948357 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:55.948359 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:55.948359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:55.948671 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:23:55.948762 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:55.949447 (+ 685us) server_negotiation.cc:299] Negotiation successful
0928 12:23:55.949639 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20260928 12:23:55.950207 17497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f898614ae6ca4a01bb7498931e120f57" candidate_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "979f8823ef2040a283f82ad07e1ed1ca" is_pre_election: true
I20260928 12:23:55.950397 17497 raft_consensus.cc:2454] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d798223e9d4c49b4b8dba3bd83defdcc in term 1.
I20260928 12:23:55.950699 17164 leader_election.cc:302] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 979f8823ef2040a283f82ad07e1ed1ca, d798223e9d4c49b4b8dba3bd83defdcc; no voters: 61304af90f2f400092fda93d54f75dc9
I20260928 12:23:55.950850 17551 raft_consensus.cc:2781] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 12:23:55.950928 17551 raft_consensus.cc:482] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:23:55.950951 17551 raft_consensus.cc:3037] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc [term 1 FOLLOWER]: Advancing to term 2
I20260928 12:23:55.952082 17551 raft_consensus.cc:504] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979f8823ef2040a283f82ad07e1ed1ca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45021 } } peers { permanent_uuid: "61304af90f2f400092fda93d54f75dc9" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34035 } } peers { permanent_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42581 } }
I20260928 12:23:55.952236 17551 leader_election.cc:288] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc [CANDIDATE]: Term 2 election: Requested vote from peers 979f8823ef2040a283f82ad07e1ed1ca (127.2.56.3:45021), 61304af90f2f400092fda93d54f75dc9 (127.2.56.2:34035)
I20260928 12:23:55.952471 17497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f898614ae6ca4a01bb7498931e120f57" candidate_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "979f8823ef2040a283f82ad07e1ed1ca"
I20260928 12:23:55.952589 17497 raft_consensus.cc:3037] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca [term 1 FOLLOWER]: Advancing to term 2
W20260928 12:23:55.952641 17161 leader_election.cc:341] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 61304af90f2f400092fda93d54f75dc9 (127.2.56.2:34035): Not found: Tablet not found: f898614ae6ca4a01bb7498931e120f57
I20260928 12:23:55.953908 17497 raft_consensus.cc:2454] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d798223e9d4c49b4b8dba3bd83defdcc in term 2.
I20260928 12:23:55.954108 17164 leader_election.cc:302] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 979f8823ef2040a283f82ad07e1ed1ca, d798223e9d4c49b4b8dba3bd83defdcc; no voters: 61304af90f2f400092fda93d54f75dc9
I20260928 12:23:55.954277 17551 raft_consensus.cc:2781] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc [term 2 FOLLOWER]: Leader election won for term 2
I20260928 12:23:55.954439 17551 raft_consensus.cc:686] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc [term 2 LEADER]: Becoming Leader. State: Replica: d798223e9d4c49b4b8dba3bd83defdcc, State: Running, Role: LEADER
I20260928 12:23:55.954559 17551 consensus_queue.cc:237] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979f8823ef2040a283f82ad07e1ed1ca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45021 } } peers { permanent_uuid: "61304af90f2f400092fda93d54f75dc9" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34035 } } peers { permanent_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42581 } }
I20260928 12:23:55.954826 17552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:55.954656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.1:55681)
0928 12:23:55.954748 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:55.954794 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 12:23:56.360554 17497 raft_consensus.cc:1261] T f898614ae6ca4a01bb7498931e120f57 P 979f8823ef2040a283f82ad07e1ed1ca [term 2 FOLLOWER]: Refusing update from remote peer d798223e9d4c49b4b8dba3bd83defdcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260928 12:23:56.361104 17551 consensus_queue.cc:1035] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "979f8823ef2040a283f82ad07e1ed1ca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260928 12:23:56.362699 17559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:56.362443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.3:60263)
0928 12:23:56.362579 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:56.362628 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":67,"threads_started":1}
I20260928 12:23:56.363678 17552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:56.363534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.1:32927)
0928 12:23:56.363583 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:56.363623 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 12:23:56.363703 17161 consensus_peers.cc:487] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc -> Peer 61304af90f2f400092fda93d54f75dc9 (127.2.56.2:34035): Couldn't send request to peer 61304af90f2f400092fda93d54f75dc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f898614ae6ca4a01bb7498931e120f57. This is attempt 1: this message will repeat every 5th retry.
I20260928 12:23:56.364020 17559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:56.363821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.3:45263)
0928 12:23:56.363871 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:56.363900 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 12:23:56.474156 17563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:56.473840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.2:47221)
0928 12:23:56.474029 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:56.474090 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":94,"threads_started":1}
I20260928 12:23:56.769088 17564 ts_tablet_manager.cc:927] T f898614ae6ca4a01bb7498931e120f57 P 61304af90f2f400092fda93d54f75dc9: Initiating tablet copy from peer d798223e9d4c49b4b8dba3bd83defdcc (127.2.56.1:42581)
I20260928 12:23:56.769397 17564 tablet_copy_client.cc:316] T f898614ae6ca4a01bb7498931e120f57 P 61304af90f2f400092fda93d54f75dc9: tablet copy: Beginning tablet copy session from remote peer at address 127.2.56.1:42581
Sep 28 12:23:56 dist-test-slave-920p krb5kdc[15979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598234, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for kudu/127.2.56.1@KRBTEST.COM
I20260928 12:23:56.783735 17563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:56.769771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:42581 (local address 127.2.56.2:58433)
0928 12:23:56.769825 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:56.769847 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 12:23:56.769919 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:23:56.770234 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:23:56.770238 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:23:56.770525 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:23:56.770608 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:56.770610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:56.771486 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:56.771486 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:56.772100 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:23:56.772102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:23:56.772565 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:23:56.772566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:23:56.772634 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:56.778985 (+ 6351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:23:56.778990 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:23:56.780593 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:23:56.782457 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:56.782464 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:56.782466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:56.782691 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:56.782956 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:23:56.782956 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:23:56.782956 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:23:56.783011 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:23:56.783387 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:23:56.783394 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:23:56.783459 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 12:23:56.783540 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 12:23:56.783647 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 12:23:56.783751 17566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:56.769880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:58433 (local address 127.2.56.1:42581)
0928 12:23:56.770057 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 12:23:56.770064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 12:23:56.770077 (+ 13us) server_negotiation.cc:407] Connection header received
0928 12:23:56.770111 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:23:56.770113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:23:56.770130 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:23:56.770216 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:23:56.770689 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:56.771408 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:56.772176 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:23:56.772490 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:23:56.772544 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:23:56.780728 (+ 8184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:23:56.780746 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:23:56.780746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:23:56.780757 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:23:56.782356 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:56.782802 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:56.782806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:56.782806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:56.782884 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:23:56.783074 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:23:56.783074 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:23:56.783075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:23:56.783306 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:23:56.783386 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 12:23:56.783579 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 12:23:56.783662 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":100,"threads_started":1}
I20260928 12:23:56.784255 17249 tablet_copy_service.cc:140] P d798223e9d4c49b4b8dba3bd83defdcc: Received BeginTabletCopySession request for tablet f898614ae6ca4a01bb7498931e120f57 from peer 61304af90f2f400092fda93d54f75dc9 ({username='kudu', principal='kudu/127.2.56.2@KRBTEST.COM'} at 127.2.56.2:58433)
I20260928 12:23:56.784327 17249 tablet_copy_service.cc:161] P d798223e9d4c49b4b8dba3bd83defdcc: Beginning new tablet copy session on tablet f898614ae6ca4a01bb7498931e120f57 from peer 61304af90f2f400092fda93d54f75dc9 at {username='kudu', principal='kudu/127.2.56.2@KRBTEST.COM'} at 127.2.56.2:58433: session id = 61304af90f2f400092fda93d54f75dc9-f898614ae6ca4a01bb7498931e120f57
I20260928 12:23:56.784866 17249 tablet_copy_source_session.cc:205] T f898614ae6ca4a01bb7498931e120f57 P d798223e9d4c49b4b8dba3bd83defdcc: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 12:23:56.786015 17564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f898614ae6ca4a01bb7498931e120f57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:23:56.788882 17564 tablet_copy_client.cc:791] T f898614ae6ca4a01bb7498931e120f57 P 61304af90f2f400092fda93d54f75dc9: tablet copy: Starting download of 0 data blocks...
I20260928 12:23:56.789036 17564 tablet_copy_client.cc:655] T f898614ae6ca4a01bb7498931e120f57 P 61304af90f2f400092fda93d54f75dc9: tablet copy: Starting download of 1 WAL segments...
I20260928 12:23:56.790169 17564 tablet_copy_client.cc:531] T f898614ae6ca4a01bb7498931e120f57 P 61304af90f2f400092fda93d54f75dc9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 12:23:56.790906 17564 tablet_bootstrap.cc:492] T f898614ae6ca4a01bb7498931e120f57 P 61304af90f2f400092fda93d54f75dc9: Bootstrap starting.
I20260928 12:23:56.792261 17564 log.cc:824] T f898614ae6ca4a01bb7498931e120f57 P 61304af90f2f400092fda93d54f75dc9: Log is configured to *not* fsync() on all Append() calls
I20260928 12:23:56.795317 17564 tablet_bootstrap.cc:492] T f898614ae6ca4a01bb7498931e120f57 P 61304af90f2f400092fda93d54f75dc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:23:56.795567 17564 tablet_bootstrap.cc:492] T f898614ae6ca4a01bb7498931e120f57 P 61304af90f2f400092fda93d54f75dc9: Bootstrap complete.
I20260928 12:23:56.795703 17564 ts_tablet_manager.cc:1397] T f898614ae6ca4a01bb7498931e120f57 P 61304af90f2f400092fda93d54f75dc9: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260928 12:23:56.796715 17564 raft_consensus.cc:348] T f898614ae6ca4a01bb7498931e120f57 P 61304af90f2f400092fda93d54f75dc9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979f8823ef2040a283f82ad07e1ed1ca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45021 } } peers { permanent_uuid: "61304af90f2f400092fda93d54f75dc9" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34035 } } peers { permanent_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42581 } }
I20260928 12:23:56.796878 17564 raft_consensus.cc:729] T f898614ae6ca4a01bb7498931e120f57 P 61304af90f2f400092fda93d54f75dc9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61304af90f2f400092fda93d54f75dc9, State: Initialized, Role: FOLLOWER
I20260928 12:23:56.796999 17564 consensus_queue.cc:260] T f898614ae6ca4a01bb7498931e120f57 P 61304af90f2f400092fda93d54f75dc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979f8823ef2040a283f82ad07e1ed1ca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45021 } } peers { permanent_uuid: "61304af90f2f400092fda93d54f75dc9" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34035 } } peers { permanent_uuid: "d798223e9d4c49b4b8dba3bd83defdcc" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 42581 } }
I20260928 12:23:56.797312 17564 ts_tablet_manager.cc:1428] T f898614ae6ca4a01bb7498931e120f57 P 61304af90f2f400092fda93d54f75dc9: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260928 12:23:56.797646 17563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:56.797518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.2:53615)
0928 12:23:56.797584 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:56.797615 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 12:23:56.797765 17249 tablet_copy_service.cc:342] P d798223e9d4c49b4b8dba3bd83defdcc: Request end of tablet copy session 61304af90f2f400092fda93d54f75dc9-f898614ae6ca4a01bb7498931e120f57 received from {username='kudu', principal='kudu/127.2.56.2@KRBTEST.COM'} at 127.2.56.2:58433
I20260928 12:23:56.797863 17249 tablet_copy_service.cc:433] P d798223e9d4c49b4b8dba3bd83defdcc: ending tablet copy session 61304af90f2f400092fda93d54f75dc9-f898614ae6ca4a01bb7498931e120f57 on tablet f898614ae6ca4a01bb7498931e120f57 with peer 61304af90f2f400092fda93d54f75dc9
I20260928 12:23:56.797971 17563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:56.797892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.2:35499)
0928 12:23:56.797930 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:56.797948 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 12:23:57.268705 17570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:57.268417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.1:54941)
0928 12:23:57.268553 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:57.268638 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":64,"threads_started":1}
I20260928 12:23:57.268651 17363 raft_consensus.cc:1203] T f898614ae6ca4a01bb7498931e120f57 P 61304af90f2f400092fda93d54f75dc9 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260928 12:23:57.269234 17563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:57.269099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.2:34029)
0928 12:23:57.269152 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:57.269188 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 12:23:57.364746 17571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:57.364498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.3:55685)
0928 12:23:57.364644 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:57.364686 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":67,"threads_started":1}
I20260928 12:23:58.269580 17572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:58.269237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.1:34809)
0928 12:23:58.269434 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:58.269491 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":100,"threads_started":1}
I20260928 12:23:58.269868 17573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:58.269622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.2:56795)
0928 12:23:58.269775 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:58.269814 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":67,"threads_started":1}
I20260928 12:23:58.365504 17574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:58.365241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.3:56667)
0928 12:23:58.365414 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:58.365454 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":118,"threads_started":1}
I20260928 12:23:59.270633 17576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:59.270146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.1:34185)
0928 12:23:59.270415 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:59.270566 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":120,"threads_started":1}
I20260928 12:23:59.270761 17577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:59.270543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.2:57259)
0928 12:23:59.270672 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:59.270711 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260928 12:23:59.366441 17578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:23:59.366100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.3:52493)
0928 12:23:59.366327 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 12:23:59.366388 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":110,"threads_started":1}
I20260928 12:24:00.271484 17579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:00.271244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.1:50955)
0928 12:24:00.271391 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:00.271428 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":78,"threads_started":1}
I20260928 12:24:00.271673 17580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:00.271317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.2:54443)
0928 12:24:00.271529 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:00.271604 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":134,"threads_started":1}
I20260928 12:24:00.271903 17579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:00.271801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.1:39535)
0928 12:24:00.271855 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:00.271880 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 12:24:00.367354 17581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:00.367011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.3:49579)
0928 12:24:00.367196 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:00.367277 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":71,"threads_started":1}
I20260928 12:24:01.272464 17582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:01.272188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.2:51891)
0928 12:24:01.272355 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:01.272411 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":94,"threads_started":1}
I20260928 12:24:01.272557 17583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:01.272339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.1:37733)
0928 12:24:01.272460 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:01.272493 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":57,"threads_started":1}
I20260928 12:24:01.368273 17584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:01.367970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.3:50049)
0928 12:24:01.368162 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:01.368209 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":101,"threads_started":1}
I20260928 12:24:02.273362 17585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:02.273033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.2:45249)
0928 12:24:02.273240 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:02.273304 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":82,"threads_started":1}
I20260928 12:24:02.273356 17586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:02.273063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.1:40387)
0928 12:24:02.273220 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:02.273285 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":73,"threads_started":1}
I20260928 12:24:02.369107 17587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:02.368815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.3:47671)
0928 12:24:02.369007 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:02.369056 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":94,"threads_started":1}
I20260928 12:24:03.274096 17589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:03.273832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.2:51041)
0928 12:24:03.274002 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:03.274038 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":78,"threads_started":1}
I20260928 12:24:03.274127 17590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:03.273822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.1:59357)
0928 12:24:03.274017 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:03.274065 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":82,"threads_started":1}
I20260928 12:24:03.369982 17591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:03.369670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.3:53625)
0928 12:24:03.369839 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:03.369900 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":88,"threads_started":1}
I20260928 12:24:04.275005 17593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:04.274699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.1:51925)
0928 12:24:04.274907 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:04.274947 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":95,"threads_started":1}
I20260928 12:24:04.275005 17592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:04.274687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.2:46985)
0928 12:24:04.274888 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:04.274940 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":101,"threads_started":1}
I20260928 12:24:04.370893 17594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:04.370546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.3:33923)
0928 12:24:04.370762 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:04.370820 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":69,"threads_started":1}
I20260928 12:24:05.275959 17596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:05.275655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.1:39149)
0928 12:24:05.275850 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:05.275898 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":87,"threads_started":1}
I20260928 12:24:05.275996 17595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:05.275655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.2:40413)
0928 12:24:05.275851 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:05.275914 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":85,"threads_started":1}
I20260928 12:24:05.371903 17597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:05.371615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.3:53727)
0928 12:24:05.371787 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:05.371851 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":85,"threads_started":1}
I20260928 12:24:06.276736 17598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:06.276530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.2:50107)
0928 12:24:06.276651 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:06.276685 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260928 12:24:06.276798 17599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:06.276523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.1:33537)
0928 12:24:06.276662 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:06.276746 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":70,"threads_started":1}
I20260928 12:24:06.372709 17600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:06.372453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.3:46423)
0928 12:24:06.372612 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:06.372656 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":83,"threads_started":1}
I20260928 12:24:07.277522 17601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:07.277235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.1:43539)
0928 12:24:07.277433 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:07.277473 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":124,"threads_started":1}
I20260928 12:24:07.277554 17602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:07.277283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.2:58387)
0928 12:24:07.277454 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:07.277494 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":78,"threads_started":1}
I20260928 12:24:07.373593 17603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:07.373255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.3:58905)
0928 12:24:07.373487 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:07.373541 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":107,"threads_started":1}
I20260928 12:24:08.278218 17604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:08.278008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.1:33823)
0928 12:24:08.278130 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:08.278168 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":64,"threads_started":1}
I20260928 12:24:08.278332 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:08.278067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.2:51801)
0928 12:24:08.278231 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:08.278264 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":98,"threads_started":1}
I20260928 12:24:08.374536 17606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:08.374215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.3:46409)
0928 12:24:08.374425 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:08.374485 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":114,"threads_started":1}
I20260928 12:24:09.279017 17607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:09.278702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.1:32835)
0928 12:24:09.278884 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:09.278945 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":98,"threads_started":1}
I20260928 12:24:09.279096 17608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:09.278855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.2:44311)
0928 12:24:09.278967 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:09.279005 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":51,"threads_started":1}
I20260928 12:24:09.375321 17609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:09.375065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:39297 (local address 127.2.56.3:36875)
0928 12:24:09.375203 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:09.375256 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":68,"threads_started":1}
I20260928 12:24:09.616726 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 17116
I20260928 12:24:09.621408 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 17279
I20260928 12:24:09.626338 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 17412
2026-09-28T12:24:09Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18054 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:24:09 dist-test-slave-920p krb5kdc[17612](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:24:09 dist-test-slave-920p krb5kdc[17612](info): set up 2 sockets
Sep 28 12:24:09 dist-test-slave-920p krb5kdc[17612](info): commencing operation
krb5kdc: starting...
W20260928 12:24:10.597252 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:24:10 dist-test-slave-920p krb5kdc[17612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:24:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:24:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
I20260928 12:24:10.712499 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:45011
--webserver_interface=127.2.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:37705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:45011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:24:10.791126 17625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:24:10.791329 17625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:24:10.791369 17625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:24:10.792510 17625 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 12:24:10.792552 17625 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 12:24:10.792569 17625 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 12:24:10.792582 17625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:24:10.792599 17625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:24:10.792624 17625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:24:10.792647 17625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:24:10.792670 17625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:24:10.794191 17625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:37705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:45011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:45011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:24:10.794436 17625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:24:10.794634 17625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:24:10.799139 17631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:24:10.799139 17633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:24:10.799162 17630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:24:10.799288 17625 server_base.cc:1034] running on GCE node
I20260928 12:24:10.799587 17625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:24:10.799846 17625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:24:10.801012 17625 hybrid_clock.cc:648] HybridClock initialized: now 1790598250800985 us; error 44 us; skew 500 ppm
Sep 28 12:24:10 dist-test-slave-920p krb5kdc[17612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598250, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:24:10.803328 17625 init.cc:377] Logged in from keytab as kudu/127.2.56.62@KRBTEST.COM (short username kudu)
I20260928 12:24:10.803750 17625 webserver.cc:460] Webserver started at http://127.2.56.62:36613/ using document root <none> and password file <none>
I20260928 12:24:10.803929 17625 fs_manager.cc:359] Metadata directory not provided
I20260928 12:24:10.803966 17625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:24:10.804054 17625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:24:10.805020 17625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "520b64e7c6ac4a4a88448716abebfbed"
format_stamp: "Formatted at 2026-09-28 12:24:10 on dist-test-slave-920p"
server_key: "56b5f746a078f88383a08f96a19eeb25"
server_key_iv: "00ce8be1862b53c60c1280b42ee36e02"
server_key_version: "encryptionkey@0"
I20260928 12:24:10.805356 17625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "520b64e7c6ac4a4a88448716abebfbed"
format_stamp: "Formatted at 2026-09-28 12:24:10 on dist-test-slave-920p"
server_key: "56b5f746a078f88383a08f96a19eeb25"
server_key_iv: "00ce8be1862b53c60c1280b42ee36e02"
server_key_version: "encryptionkey@0"
I20260928 12:24:10.806588 17625 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:24:10.807241 17640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:24:10.807410 17625 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:24:10.807493 17625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "520b64e7c6ac4a4a88448716abebfbed"
format_stamp: "Formatted at 2026-09-28 12:24:10 on dist-test-slave-920p"
server_key: "56b5f746a078f88383a08f96a19eeb25"
server_key_iv: "00ce8be1862b53c60c1280b42ee36e02"
server_key_version: "encryptionkey@0"
I20260928 12:24:10.807559 17625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:24:10.827188 17625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:24:10.828953 17625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:24:10.829072 17625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:24:10.832872 17625 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:45011
I20260928 12:24:10.832913 17692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:45011 every 8 connection(s)
I20260928 12:24:10.833222 17625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:24:10.834053 17693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:24:10.836228 17693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed: Bootstrap starting.
I20260928 12:24:10.836776 17693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed: Neither blocks nor log segments found. Creating new log.
I20260928 12:24:10.837007 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 17625
I20260928 12:24:10.837042 17693 log.cc:824] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed: Log is configured to *not* fsync() on all Append() calls
I20260928 12:24:10.837085 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:24:10.837285 2272 external_mini_cluster.cc:1428] Setting key 7c9fdd6c8a52d2a9a98aa5bc8bb4c10f
I20260928 12:24:10.837744 17693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed: No bootstrap required, opened a new log
I20260928 12:24:10.839351 17693 raft_consensus.cc:348] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520b64e7c6ac4a4a88448716abebfbed" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 45011 } }
I20260928 12:24:10.839479 17693 raft_consensus.cc:374] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:24:10.839509 17693 raft_consensus.cc:729] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 520b64e7c6ac4a4a88448716abebfbed, State: Initialized, Role: FOLLOWER
I20260928 12:24:10.839677 17693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520b64e7c6ac4a4a88448716abebfbed" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 45011 } }
I20260928 12:24:10.839764 17693 raft_consensus.cc:388] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:24:10.839804 17693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:24:10.839850 17693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:24:10.840729 17693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520b64e7c6ac4a4a88448716abebfbed" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 45011 } }
I20260928 12:24:10.840814 17693 leader_election.cc:302] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 520b64e7c6ac4a4a88448716abebfbed; no voters:
I20260928 12:24:10.840955 17693 leader_election.cc:288] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:24:10.840986 17698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:24:10.841118 17698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [term 1 LEADER]: Becoming Leader. State: Replica: 520b64e7c6ac4a4a88448716abebfbed, State: Running, Role: LEADER
I20260928 12:24:10.841198 17693 sys_catalog.cc:564] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:24:10.841286 17698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520b64e7c6ac4a4a88448716abebfbed" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 45011 } }
I20260928 12:24:10.841663 17698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 520b64e7c6ac4a4a88448716abebfbed. Latest consensus state: current_term: 1 leader_uuid: "520b64e7c6ac4a4a88448716abebfbed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520b64e7c6ac4a4a88448716abebfbed" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 45011 } } }
I20260928 12:24:10.841991 17698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [sys.catalog]: This master's current role is: LEADER
I20260928 12:24:10.841925 17700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "520b64e7c6ac4a4a88448716abebfbed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520b64e7c6ac4a4a88448716abebfbed" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 45011 } } }
I20260928 12:24:10.842372 17700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [sys.catalog]: This master's current role is: LEADER
I20260928 12:24:10.842561 17707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:24:10.843286 17707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:24:10.844869 17707 catalog_manager.cc:1347] Generated new cluster ID: 51b36946c0ec4f13a19d46635ddc523f
I20260928 12:24:10.844918 17707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:24:10 dist-test-slave-920p krb5kdc[17612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:24:10.848518 17697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:10.837748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40018 (local address 127.2.56.62:45011)
0928 12:24:10.837918 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 12:24:10.837924 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:24:10.837947 (+ 23us) server_negotiation.cc:407] Connection header received
0928 12:24:10.839349 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:24:10.839356 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:24:10.839405 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:24:10.839535 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:24:10.840281 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:10.841439 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:10.842490 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:10.843031 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:10.843128 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:10.845559 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:24:10.845564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:24:10.845564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:24:10.845575 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:24:10.847067 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:24:10.847384 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:24:10.847388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:24:10.847388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:24:10.847486 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:24:10.847658 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:24:10.847658 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:24:10.847659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:24:10.847929 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:24:10.848060 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 12:24:10.848165 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 12:24:10.848299 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":64,"threads_started":1}
I20260928 12:24:10.857345 17707 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:24:10.857895 17707 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:24:10.866972 17707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed: Generated new TSK 0
I20260928 12:24:10.867146 17707 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab.
Entry for principal kudu/127.2.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab.
I20260928 12:24:10.900645 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.1:0
--local_ip_for_outbound_sockets=127.2.56.1
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:45011
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:37705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:24:10.988868 17719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:24:10.989060 17719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:24:10.989081 17719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:24:10.990213 17719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:24:10.990254 17719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:24:10.990278 17719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.1
I20260928 12:24:10.991673 17719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:37705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:45011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.1
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:24:10.991907 17719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:24:10.992121 17719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:24:10.998150 17725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:24:10.998116 17724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:24:10.998432 17727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:24:10.998796 17719 server_base.cc:1034] running on GCE node
I20260928 12:24:10.999061 17719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:24:10.999327 17719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:24:11.000466 17719 hybrid_clock.cc:648] HybridClock initialized: now 1790598251000454 us; error 25 us; skew 500 ppm
Sep 28 12:24:10 dist-test-slave-920p krb5kdc[17612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598250, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:24:11.002815 17719 init.cc:377] Logged in from keytab as kudu/127.2.56.1@KRBTEST.COM (short username kudu)
I20260928 12:24:11.003197 17719 webserver.cc:460] Webserver started at http://127.2.56.1:38051/ using document root <none> and password file <none>
I20260928 12:24:11.003367 17719 fs_manager.cc:359] Metadata directory not provided
I20260928 12:24:11.003404 17719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:24:11.003489 17719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:24:11.004480 17719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/data/instance:
uuid: "e3acf94f9ac746f7b4d817b2eb4d2819"
format_stamp: "Formatted at 2026-09-28 12:24:10 on dist-test-slave-920p"
server_key: "65e27cb6ff43496afc107dbdd9dd08ef"
server_key_iv: "0434006bac5b175cf3fc786a33059582"
server_key_version: "encryptionkey@0"
I20260928 12:24:11.004757 17719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance:
uuid: "e3acf94f9ac746f7b4d817b2eb4d2819"
format_stamp: "Formatted at 2026-09-28 12:24:10 on dist-test-slave-920p"
server_key: "65e27cb6ff43496afc107dbdd9dd08ef"
server_key_iv: "0434006bac5b175cf3fc786a33059582"
server_key_version: "encryptionkey@0"
I20260928 12:24:11.005939 17719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:24:11.006569 17734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:24:11.006704 17719 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:24:11.006776 17719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/wal
uuid: "e3acf94f9ac746f7b4d817b2eb4d2819"
format_stamp: "Formatted at 2026-09-28 12:24:10 on dist-test-slave-920p"
server_key: "65e27cb6ff43496afc107dbdd9dd08ef"
server_key_iv: "0434006bac5b175cf3fc786a33059582"
server_key_version: "encryptionkey@0"
I20260928 12:24:11.006847 17719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:24:11.023909 17719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:24:11.025879 17719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:24:11.025996 17719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:24:11.026219 17719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:24:11.026494 17719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:24:11.026525 17719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:24:11.026544 17719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:24:11.026567 17719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:24:11.031730 17719 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.1:37109
I20260928 12:24:11.031803 17847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.1:37109 every 8 connection(s)
I20260928 12:24:11.032068 17719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
I20260928 12:24:11.035807 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 17719
I20260928 12:24:11.035928 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance
I20260928 12:24:11.036119 2272 external_mini_cluster.cc:1428] Setting key 4fc8569cd5696340d63a5797f3f722c5
Sep 28 12:24:11 dist-test-slave-920p krb5kdc[17612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598250, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:24:11.041220 17697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:11.032708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:56495 (local address 127.2.56.62:45011)
0928 12:24:11.032780 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 12:24:11.032782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:24:11.032913 (+ 131us) server_negotiation.cc:407] Connection header received
0928 12:24:11.033393 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:24:11.033394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:24:11.033407 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:24:11.033455 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:24:11.034312 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:11.034801 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:11.035804 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:11.036158 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:11.036233 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:11.038489 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:24:11.038490 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:24:11.038491 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:24:11.038501 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:24:11.039749 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:24:11.040220 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:24:11.040221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:24:11.040222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:24:11.040274 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:24:11.040528 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:24:11.040529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:24:11.040529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:24:11.040630 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:24:11.040677 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 12:24:11.041011 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 12:24:11.041148 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 12:24:11.041401 17850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:11.032674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.1:56495)
0928 12:24:11.032802 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:11.032837 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 12:24:11.033323 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:24:11.033504 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:24:11.033507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:24:11.034044 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:24:11.034241 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:24:11.034245 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:24:11.034881 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:24:11.034882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:24:11.035715 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:24:11.035718 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:24:11.036307 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:24:11.036309 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:24:11.036448 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:11.036903 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:24:11.036914 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:24:11.038404 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:24:11.039839 (+ 1435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:24:11.039847 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:24:11.039849 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:24:11.040139 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:24:11.040335 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:24:11.040335 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:24:11.040336 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:24:11.040468 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:24:11.040686 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:24:11.040691 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:24:11.040865 (+ 174us) client_negotiation.cc:769] Sending connection context
0928 12:24:11.040977 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 12:24:11.041143 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":67,"threads_started":1}
I20260928 12:24:11.041968 17848 heartbeater.cc:347] Connected to a master server at 127.2.56.62:45011
I20260928 12:24:11.042068 17848 heartbeater.cc:464] Registering TS with master...
I20260928 12:24:11.042256 17848 heartbeater.cc:511] Master 127.2.56.62:45011 requested a full tablet report, sending...
I20260928 12:24:11.042742 17657 ts_manager.cc:194] Registered new tserver with Master: e3acf94f9ac746f7b4d817b2eb4d2819 (127.2.56.1:37109)
I20260928 12:24:11.043372 17657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.1@KRBTEST.COM'} at 127.2.56.1:56495
WARNING: no policy specified for kudu/127.2.56.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab.
Entry for principal kudu/127.2.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab.
I20260928 12:24:11.060895 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.2:0
--local_ip_for_outbound_sockets=127.2.56.2
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:45011
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:37705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:24:11.137852 17853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:24:11.138034 17853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:24:11.138055 17853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:24:11.139153 17853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:24:11.139191 17853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:24:11.139222 17853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.2
I20260928 12:24:11.140635 17853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:37705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:45011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.2
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:24:11.140851 17853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:24:11.141057 17853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:24:11.145993 17861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:24:11.145991 17858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:24:11.146016 17853 server_base.cc:1034] running on GCE node
W20260928 12:24:11.145988 17859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:24:11.146432 17853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:24:11.146656 17853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:24:11.147785 17853 hybrid_clock.cc:648] HybridClock initialized: now 1790598251147768 us; error 34 us; skew 500 ppm
Sep 28 12:24:11 dist-test-slave-920p krb5kdc[17612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598251, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:24:11.150100 17853 init.cc:377] Logged in from keytab as kudu/127.2.56.2@KRBTEST.COM (short username kudu)
I20260928 12:24:11.150512 17853 webserver.cc:460] Webserver started at http://127.2.56.2:39979/ using document root <none> and password file <none>
I20260928 12:24:11.150730 17853 fs_manager.cc:359] Metadata directory not provided
I20260928 12:24:11.150780 17853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:24:11.150900 17853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:24:11.151899 17853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/data/instance:
uuid: "7e959ec3b76e4e31850d8986edb31f10"
format_stamp: "Formatted at 2026-09-28 12:24:11 on dist-test-slave-920p"
server_key: "b30db4177d546d8e45c6c86e9c6f3fce"
server_key_iv: "ec216594c0f1bf5c511832a81f62cb9c"
server_key_version: "encryptionkey@0"
I20260928 12:24:11.152205 17853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance:
uuid: "7e959ec3b76e4e31850d8986edb31f10"
format_stamp: "Formatted at 2026-09-28 12:24:11 on dist-test-slave-920p"
server_key: "b30db4177d546d8e45c6c86e9c6f3fce"
server_key_iv: "ec216594c0f1bf5c511832a81f62cb9c"
server_key_version: "encryptionkey@0"
I20260928 12:24:11.153470 17853 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:24:11.154103 17868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:24:11.154246 17853 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:24:11.154325 17853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/wal
uuid: "7e959ec3b76e4e31850d8986edb31f10"
format_stamp: "Formatted at 2026-09-28 12:24:11 on dist-test-slave-920p"
server_key: "b30db4177d546d8e45c6c86e9c6f3fce"
server_key_iv: "ec216594c0f1bf5c511832a81f62cb9c"
server_key_version: "encryptionkey@0"
I20260928 12:24:11.154399 17853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:24:11.162696 17853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:24:11.164354 17853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:24:11.164486 17853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:24:11.164673 17853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:24:11.164953 17853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:24:11.164986 17853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:24:11.165014 17853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:24:11.165036 17853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:24:11.170498 17853 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.2:45547
I20260928 12:24:11.170557 17981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.2:45547 every 8 connection(s)
I20260928 12:24:11.170817 17853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
I20260928 12:24:11.175041 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 17853
I20260928 12:24:11.175132 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance
I20260928 12:24:11.175554 2272 external_mini_cluster.cc:1428] Setting key 99279e3d577e47a46fece244b64515e4
Sep 28 12:24:11 dist-test-slave-920p krb5kdc[17612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598251, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:24:11.179597 17697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:11.171450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:39035 (local address 127.2.56.62:45011)
0928 12:24:11.171505 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 12:24:11.171507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:24:11.171672 (+ 165us) server_negotiation.cc:407] Connection header received
0928 12:24:11.172161 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:24:11.172162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:24:11.172175 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:24:11.172226 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:24:11.173122 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:11.173676 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:11.174611 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:11.174812 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:11.174893 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:11.176948 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:24:11.176949 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:24:11.176950 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:24:11.176958 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:24:11.178281 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:24:11.178673 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:24:11.178674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:24:11.178674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:24:11.178737 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:24:11.178939 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:24:11.178939 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:24:11.178940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:24:11.179063 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:24:11.179128 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 12:24:11.179408 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 12:24:11.179503 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 12:24:11.179633 17984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:11.171419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.2:39035)
0928 12:24:11.171562 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:11.171595 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 12:24:11.172064 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:24:11.172288 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:24:11.172291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:24:11.172842 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:24:11.173028 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:24:11.173032 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:24:11.173758 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:24:11.173759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:24:11.174529 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:24:11.174532 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:24:11.174879 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:24:11.174880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:24:11.175003 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:11.175411 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:24:11.175419 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:24:11.176865 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:24:11.178356 (+ 1491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:24:11.178362 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:24:11.178364 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:24:11.178592 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:24:11.178797 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:24:11.178797 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:24:11.178798 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:24:11.178888 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:24:11.179145 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:24:11.179150 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:24:11.179274 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 12:24:11.179365 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 12:24:11.179482 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
I20260928 12:24:11.180088 17982 heartbeater.cc:347] Connected to a master server at 127.2.56.62:45011
I20260928 12:24:11.180186 17982 heartbeater.cc:464] Registering TS with master...
I20260928 12:24:11.180384 17982 heartbeater.cc:511] Master 127.2.56.62:45011 requested a full tablet report, sending...
I20260928 12:24:11.180774 17657 ts_manager.cc:194] Registered new tserver with Master: 7e959ec3b76e4e31850d8986edb31f10 (127.2.56.2:45547)
I20260928 12:24:11.181126 17657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.2@KRBTEST.COM'} at 127.2.56.2:39035
WARNING: no policy specified for kudu/127.2.56.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab.
Entry for principal kudu/127.2.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab.
I20260928 12:24:11.198160 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.3:0
--local_ip_for_outbound_sockets=127.2.56.3
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:45011
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:37705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:24:11.277213 17987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:24:11.277485 17987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:24:11.277521 17987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:24:11.279170 17987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:24:11.279219 17987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:24:11.279255 17987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.3
I20260928 12:24:11.280958 17987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:37705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:45011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.3
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:24:11.281195 17987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:24:11.281462 17987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:24:11.287113 17987 server_base.cc:1034] running on GCE node
W20260928 12:24:11.287271 17995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:24:11.287385 17993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:24:11.287599 17992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:24:11.287839 17987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:24:11.288066 17987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:24:11.289207 17987 hybrid_clock.cc:648] HybridClock initialized: now 1790598251289185 us; error 43 us; skew 500 ppm
Sep 28 12:24:11 dist-test-slave-920p krb5kdc[17612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598251, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:24:11.291890 17987 init.cc:377] Logged in from keytab as kudu/127.2.56.3@KRBTEST.COM (short username kudu)
I20260928 12:24:11.292398 17987 webserver.cc:460] Webserver started at http://127.2.56.3:43413/ using document root <none> and password file <none>
I20260928 12:24:11.292634 17987 fs_manager.cc:359] Metadata directory not provided
I20260928 12:24:11.292706 17987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:24:11.292843 17987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:24:11.294283 17987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/data/instance:
uuid: "73379ec00f7842ada1c253cf0175e4f5"
format_stamp: "Formatted at 2026-09-28 12:24:11 on dist-test-slave-920p"
server_key: "239e88530eee59979c4d28502e69c083"
server_key_iv: "d262034a29f19ea725b0c006595fde87"
server_key_version: "encryptionkey@0"
I20260928 12:24:11.294703 17987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance:
uuid: "73379ec00f7842ada1c253cf0175e4f5"
format_stamp: "Formatted at 2026-09-28 12:24:11 on dist-test-slave-920p"
server_key: "239e88530eee59979c4d28502e69c083"
server_key_iv: "d262034a29f19ea725b0c006595fde87"
server_key_version: "encryptionkey@0"
I20260928 12:24:11.296358 17987 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:24:11.297196 18002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:24:11.297384 17987 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:24:11.297465 17987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/wal
uuid: "73379ec00f7842ada1c253cf0175e4f5"
format_stamp: "Formatted at 2026-09-28 12:24:11 on dist-test-slave-920p"
server_key: "239e88530eee59979c4d28502e69c083"
server_key_iv: "d262034a29f19ea725b0c006595fde87"
server_key_version: "encryptionkey@0"
I20260928 12:24:11.297533 17987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:24:11.332943 17987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:24:11.334952 17987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:24:11.335074 17987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:24:11.335338 17987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:24:11.335641 17987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:24:11.335675 17987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:24:11.335695 17987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:24:11.335707 17987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:24:11.341089 17987 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.3:41343
I20260928 12:24:11.341153 18115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.3:41343 every 8 connection(s)
I20260928 12:24:11.341468 17987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
I20260928 12:24:11.342795 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 17987
I20260928 12:24:11.342890 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance
I20260928 12:24:11.343079 2272 external_mini_cluster.cc:1428] Setting key 09b4a27924c473bdb667027a0443eaa9
Sep 28 12:24:11 dist-test-slave-920p krb5kdc[17612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598251, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:24:11.351094 17697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:11.342143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:59105 (local address 127.2.56.62:45011)
0928 12:24:11.342220 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 12:24:11.342222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:24:11.342370 (+ 148us) server_negotiation.cc:407] Connection header received
0928 12:24:11.343107 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:24:11.343108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:24:11.343121 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:24:11.343179 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:24:11.344376 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:11.345149 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:11.346390 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:11.346578 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:11.346635 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:11.348693 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:24:11.348694 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:24:11.348694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:24:11.348702 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:24:11.349951 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:24:11.350333 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:24:11.350334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:24:11.350334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:24:11.350388 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:24:11.350568 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:24:11.350569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:24:11.350569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:24:11.350668 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:24:11.350713 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 12:24:11.350969 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 12:24:11.351028 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 12:24:11.351280 18118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:11.342122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.3:59105)
0928 12:24:11.342257 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:11.342291 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 12:24:11.342986 (+ 695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:24:11.343272 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:24:11.343352 (+ 80us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:24:11.344099 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:24:11.344297 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:24:11.344302 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:24:11.345240 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:24:11.345241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:24:11.346316 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:24:11.346319 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:24:11.346650 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:24:11.346651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:24:11.346759 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:11.347258 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:24:11.347266 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:24:11.348609 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:24:11.350023 (+ 1414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:24:11.350029 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:24:11.350031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:24:11.350270 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:24:11.350436 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:24:11.350436 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:24:11.350436 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:24:11.350526 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:24:11.350708 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:24:11.350711 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:24:11.350868 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 12:24:11.350951 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 12:24:11.351071 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":71,"threads_started":1}
I20260928 12:24:11.351691 18116 heartbeater.cc:347] Connected to a master server at 127.2.56.62:45011
I20260928 12:24:11.351780 18116 heartbeater.cc:464] Registering TS with master...
I20260928 12:24:11.351943 18116 heartbeater.cc:511] Master 127.2.56.62:45011 requested a full tablet report, sending...
I20260928 12:24:11.352313 17657 ts_manager.cc:194] Registered new tserver with Master: 73379ec00f7842ada1c253cf0175e4f5 (127.2.56.3:41343)
I20260928 12:24:11.352638 17657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.3@KRBTEST.COM'} at 127.2.56.3:59105
I20260928 12:24:11.354545 2272 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:24:11.361166 17697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:11.355911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40026 (local address 127.2.56.62:45011)
0928 12:24:11.355974 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 12:24:11.355976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:24:11.356025 (+ 49us) server_negotiation.cc:407] Connection header received
0928 12:24:11.356084 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:24:11.356085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:24:11.356095 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:24:11.356142 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:24:11.356690 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:11.357219 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:11.357907 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:11.358138 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:11.358190 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:11.359160 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:24:11.359161 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:24:11.359161 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:24:11.359168 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:24:11.360125 (+ 957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:24:11.360442 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:24:11.360443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:24:11.360444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:24:11.360496 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:24:11.360648 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:24:11.360648 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:24:11.360649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:24:11.360801 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:24:11.360891 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 12:24:11.361006 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 12:24:11.361078 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 12:24:11.362638 17657 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:24:11.364141 17657 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:24:11.370256 18126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:11.366195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:41343 (local address 127.0.0.1:49220)
0928 12:24:11.366585 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:11.366624 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 12:24:11.366690 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:24:11.366958 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:24:11.366961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:24:11.366963 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:24:11.367026 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:24:11.367028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:24:11.368329 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:24:11.368330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:24:11.369370 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:24:11.369372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:24:11.369969 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:24:11.369970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:24:11.370056 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:11.370069 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 12:24:11.370129 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 12:24:11.370183 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":58,"threads_started":1}
I20260928 12:24:11.370383 18127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:11.366409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36838 (local address 127.2.56.2:45547)
0928 12:24:11.366582 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 12:24:11.366587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:24:11.366731 (+ 144us) server_negotiation.cc:407] Connection header received
0928 12:24:11.366798 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:24:11.366800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:24:11.366815 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:24:11.366891 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:24:11.367113 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:11.367889 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:11.369358 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:11.370070 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:11.370139 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:11.370188 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 12:24:11.370266 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 12:24:11.370301 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":82,"threads_started":1}
I20260928 12:24:11.370354 18125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:11.366195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:45547 (local address 127.0.0.1:36838)
0928 12:24:11.366655 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:11.366664 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:24:11.366728 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:24:11.366967 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:24:11.366968 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:24:11.366970 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:24:11.367037 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:24:11.367038 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:24:11.367978 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:24:11.367979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:24:11.369294 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:24:11.369297 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:24:11.370157 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:24:11.370158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:24:11.370210 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:11.370213 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:24:11.370257 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 12:24:11.370291 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":2,"thread_start_us":52,"threads_started":1}
I20260928 12:24:11.370725 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:11.366429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49220 (local address 127.2.56.3:41343)
0928 12:24:11.366612 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 12:24:11.366616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:24:11.366686 (+ 70us) server_negotiation.cc:407] Connection header received
0928 12:24:11.366769 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:24:11.366771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:24:11.366786 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:24:11.366855 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:24:11.367100 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:11.368216 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:11.369424 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:11.369894 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:11.369957 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:11.370014 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 12:24:11.370604 (+ 590us) server_negotiation.cc:299] Negotiation successful
0928 12:24:11.370649 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":84,"threads_started":1}
I20260928 12:24:11.371203 17916 tablet_service.cc:1467] Processing CreateTablet for tablet 004e417071894df998cf5aaf10bf4c2f (DEFAULT_TABLE table=test-table [id=5675c966d6d1452ea0aa67141f9f35f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:24:11.371352 18050 tablet_service.cc:1467] Processing CreateTablet for tablet 004e417071894df998cf5aaf10bf4c2f (DEFAULT_TABLE table=test-table [id=5675c966d6d1452ea0aa67141f9f35f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:24:11.371587 17916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 004e417071894df998cf5aaf10bf4c2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:24:11.371604 18124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:11.366205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:37109 (local address 127.0.0.1:51138)
0928 12:24:11.366773 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:11.366781 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:24:11.366825 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:24:11.367041 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:24:11.367042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:24:11.367044 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:24:11.367081 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:24:11.367083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:24:11.369351 (+ 2268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:24:11.369352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:24:11.370427 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:24:11.370430 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:24:11.371380 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:24:11.371381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:24:11.371441 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:11.371444 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:24:11.371498 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 12:24:11.371538 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":56,"threads_started":1}
I20260928 12:24:11.371680 18050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 004e417071894df998cf5aaf10bf4c2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:24:11.371855 18128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:11.366441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51138 (local address 127.2.56.1:37109)
0928 12:24:11.366614 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 12:24:11.366617 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:24:11.366827 (+ 210us) server_negotiation.cc:407] Connection header received
0928 12:24:11.366889 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:24:11.366891 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:24:11.366904 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:24:11.366969 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:24:11.367150 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:11.368223 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:11.370517 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:11.371232 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:11.371297 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:11.371347 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 12:24:11.371582 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 12:24:11.371777 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":67,"threads_started":1}
I20260928 12:24:11.374050 17782 tablet_service.cc:1467] Processing CreateTablet for tablet 004e417071894df998cf5aaf10bf4c2f (DEFAULT_TABLE table=test-table [id=5675c966d6d1452ea0aa67141f9f35f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:24:11.374307 18130 tablet_bootstrap.cc:492] T 004e417071894df998cf5aaf10bf4c2f P 7e959ec3b76e4e31850d8986edb31f10: Bootstrap starting.
I20260928 12:24:11.374372 17782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 004e417071894df998cf5aaf10bf4c2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:24:11.374872 18130 tablet_bootstrap.cc:654] T 004e417071894df998cf5aaf10bf4c2f P 7e959ec3b76e4e31850d8986edb31f10: Neither blocks nor log segments found. Creating new log.
I20260928 12:24:11.375257 18130 log.cc:824] T 004e417071894df998cf5aaf10bf4c2f P 7e959ec3b76e4e31850d8986edb31f10: Log is configured to *not* fsync() on all Append() calls
I20260928 12:24:11.375865 18130 tablet_bootstrap.cc:492] T 004e417071894df998cf5aaf10bf4c2f P 7e959ec3b76e4e31850d8986edb31f10: No bootstrap required, opened a new log
I20260928 12:24:11.375910 18130 ts_tablet_manager.cc:1397] T 004e417071894df998cf5aaf10bf4c2f P 7e959ec3b76e4e31850d8986edb31f10: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:24:11.376596 18133 tablet_bootstrap.cc:492] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819: Bootstrap starting.
I20260928 12:24:11.376865 18131 tablet_bootstrap.cc:492] T 004e417071894df998cf5aaf10bf4c2f P 73379ec00f7842ada1c253cf0175e4f5: Bootstrap starting.
I20260928 12:24:11.377192 18133 tablet_bootstrap.cc:654] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819: Neither blocks nor log segments found. Creating new log.
I20260928 12:24:11.377499 18131 tablet_bootstrap.cc:654] T 004e417071894df998cf5aaf10bf4c2f P 73379ec00f7842ada1c253cf0175e4f5: Neither blocks nor log segments found. Creating new log.
I20260928 12:24:11.377588 18130 raft_consensus.cc:348] T 004e417071894df998cf5aaf10bf4c2f P 7e959ec3b76e4e31850d8986edb31f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e959ec3b76e4e31850d8986edb31f10" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 45547 } } peers { permanent_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37109 } } peers { permanent_uuid: "73379ec00f7842ada1c253cf0175e4f5" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41343 } }
I20260928 12:24:11.377737 18130 raft_consensus.cc:374] T 004e417071894df998cf5aaf10bf4c2f P 7e959ec3b76e4e31850d8986edb31f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:24:11.377776 18130 raft_consensus.cc:729] T 004e417071894df998cf5aaf10bf4c2f P 7e959ec3b76e4e31850d8986edb31f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e959ec3b76e4e31850d8986edb31f10, State: Initialized, Role: FOLLOWER
I20260928 12:24:11.377766 18133 log.cc:824] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819: Log is configured to *not* fsync() on all Append() calls
I20260928 12:24:11.377923 18130 consensus_queue.cc:260] T 004e417071894df998cf5aaf10bf4c2f P 7e959ec3b76e4e31850d8986edb31f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e959ec3b76e4e31850d8986edb31f10" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 45547 } } peers { permanent_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37109 } } peers { permanent_uuid: "73379ec00f7842ada1c253cf0175e4f5" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41343 } }
I20260928 12:24:11.378026 18131 log.cc:824] T 004e417071894df998cf5aaf10bf4c2f P 73379ec00f7842ada1c253cf0175e4f5: Log is configured to *not* fsync() on all Append() calls
I20260928 12:24:11.378182 18130 ts_tablet_manager.cc:1428] T 004e417071894df998cf5aaf10bf4c2f P 7e959ec3b76e4e31850d8986edb31f10: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:24:11.378310 17982 heartbeater.cc:503] Master 127.2.56.62:45011 was elected leader, sending a full tablet report...
I20260928 12:24:11.378499 18133 tablet_bootstrap.cc:492] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819: No bootstrap required, opened a new log
I20260928 12:24:11.378561 18133 ts_tablet_manager.cc:1397] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:24:11.380084 18133 raft_consensus.cc:348] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e959ec3b76e4e31850d8986edb31f10" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 45547 } } peers { permanent_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37109 } } peers { permanent_uuid: "73379ec00f7842ada1c253cf0175e4f5" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41343 } }
I20260928 12:24:11.380218 18133 raft_consensus.cc:374] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:24:11.380249 18133 raft_consensus.cc:729] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3acf94f9ac746f7b4d817b2eb4d2819, State: Initialized, Role: FOLLOWER
I20260928 12:24:11.380355 18133 consensus_queue.cc:260] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e959ec3b76e4e31850d8986edb31f10" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 45547 } } peers { permanent_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37109 } } peers { permanent_uuid: "73379ec00f7842ada1c253cf0175e4f5" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41343 } }
I20260928 12:24:11.380569 18133 ts_tablet_manager.cc:1428] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:24:11.380620 17848 heartbeater.cc:503] Master 127.2.56.62:45011 was elected leader, sending a full tablet report...
I20260928 12:24:11.381119 18131 tablet_bootstrap.cc:492] T 004e417071894df998cf5aaf10bf4c2f P 73379ec00f7842ada1c253cf0175e4f5: No bootstrap required, opened a new log
I20260928 12:24:11.381178 18131 ts_tablet_manager.cc:1397] T 004e417071894df998cf5aaf10bf4c2f P 73379ec00f7842ada1c253cf0175e4f5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 12:24:11.382602 18131 raft_consensus.cc:348] T 004e417071894df998cf5aaf10bf4c2f P 73379ec00f7842ada1c253cf0175e4f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e959ec3b76e4e31850d8986edb31f10" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 45547 } } peers { permanent_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37109 } } peers { permanent_uuid: "73379ec00f7842ada1c253cf0175e4f5" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41343 } }
I20260928 12:24:11.382736 18131 raft_consensus.cc:374] T 004e417071894df998cf5aaf10bf4c2f P 73379ec00f7842ada1c253cf0175e4f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:24:11.382767 18131 raft_consensus.cc:729] T 004e417071894df998cf5aaf10bf4c2f P 73379ec00f7842ada1c253cf0175e4f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73379ec00f7842ada1c253cf0175e4f5, State: Initialized, Role: FOLLOWER
I20260928 12:24:11.382871 18131 consensus_queue.cc:260] T 004e417071894df998cf5aaf10bf4c2f P 73379ec00f7842ada1c253cf0175e4f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e959ec3b76e4e31850d8986edb31f10" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 45547 } } peers { permanent_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37109 } } peers { permanent_uuid: "73379ec00f7842ada1c253cf0175e4f5" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41343 } }
I20260928 12:24:11.383096 18131 ts_tablet_manager.cc:1428] T 004e417071894df998cf5aaf10bf4c2f P 73379ec00f7842ada1c253cf0175e4f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:24:11.383136 18116 heartbeater.cc:503] Master 127.2.56.62:45011 was elected leader, sending a full tablet report...
I20260928 12:24:11.389578 18137 raft_consensus.cc:482] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:24:11.389659 18137 raft_consensus.cc:504] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e959ec3b76e4e31850d8986edb31f10" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 45547 } } peers { permanent_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37109 } } peers { permanent_uuid: "73379ec00f7842ada1c253cf0175e4f5" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41343 } }
I20260928 12:24:11.389972 18137 leader_election.cc:288] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e959ec3b76e4e31850d8986edb31f10 (127.2.56.2:45547), 73379ec00f7842ada1c253cf0175e4f5 (127.2.56.3:41343)
I20260928 12:24:11.393483 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:11.390159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:55355 (local address 127.2.56.3:41343)
0928 12:24:11.390230 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 12:24:11.390231 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:24:11.390342 (+ 111us) server_negotiation.cc:407] Connection header received
0928 12:24:11.390370 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:24:11.390371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:24:11.390380 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:24:11.390417 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:24:11.390525 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:11.391209 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:11.392576 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:11.393132 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:11.393207 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:11.393215 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 12:24:11.393384 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 12:24:11.393421 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 12:24:11.393491 18127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:11.390166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:46781 (local address 127.2.56.2:45547)
0928 12:24:11.390222 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 12:24:11.390224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:24:11.390233 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:24:11.390256 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:24:11.390257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:24:11.390269 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:24:11.390314 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:24:11.390469 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:11.391205 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:11.392576 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:11.393132 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:11.393201 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:11.393211 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 12:24:11.393403 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 12:24:11.393436 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 12:24:11.393509 17850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:11.390097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:45547 (local address 127.2.56.1:46781)
0928 12:24:11.390148 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:11.390159 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:24:11.390194 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:24:11.390322 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:24:11.390323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:24:11.390324 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:24:11.390411 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:24:11.390413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:24:11.391272 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:24:11.391272 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:24:11.392496 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:24:11.392499 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:24:11.393209 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:24:11.393211 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:24:11.393318 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:11.393320 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:24:11.393396 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 12:24:11.393426 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":47}
I20260928 12:24:11.393517 18139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:11.390111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:41343 (local address 127.2.56.1:55355)
0928 12:24:11.390279 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:11.390287 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:24:11.390335 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:24:11.390420 (+ 85us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:24:11.390420 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:24:11.390421 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:24:11.390474 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:24:11.390476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:24:11.391276 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:24:11.391277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:24:11.392496 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:24:11.392499 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:24:11.393210 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:24:11.393211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:24:11.393314 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:11.393319 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:24:11.393378 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 12:24:11.393421 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":27,"thread_start_us":74,"threads_started":1}
I20260928 12:24:11.393815 17936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004e417071894df998cf5aaf10bf4c2f" candidate_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e959ec3b76e4e31850d8986edb31f10" is_pre_election: true
I20260928 12:24:11.393885 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004e417071894df998cf5aaf10bf4c2f" candidate_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73379ec00f7842ada1c253cf0175e4f5" is_pre_election: true
I20260928 12:24:11.393981 17936 raft_consensus.cc:2454] T 004e417071894df998cf5aaf10bf4c2f P 7e959ec3b76e4e31850d8986edb31f10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3acf94f9ac746f7b4d817b2eb4d2819 in term 0.
I20260928 12:24:11.393981 18070 raft_consensus.cc:2454] T 004e417071894df998cf5aaf10bf4c2f P 73379ec00f7842ada1c253cf0175e4f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3acf94f9ac746f7b4d817b2eb4d2819 in term 0.
I20260928 12:24:11.394167 17736 leader_election.cc:302] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73379ec00f7842ada1c253cf0175e4f5, e3acf94f9ac746f7b4d817b2eb4d2819; no voters:
I20260928 12:24:11.394299 18137 raft_consensus.cc:2781] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:24:11.394351 18137 raft_consensus.cc:482] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:24:11.394371 18137 raft_consensus.cc:3037] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:24:11.394982 18137 raft_consensus.cc:504] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e959ec3b76e4e31850d8986edb31f10" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 45547 } } peers { permanent_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37109 } } peers { permanent_uuid: "73379ec00f7842ada1c253cf0175e4f5" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41343 } }
I20260928 12:24:11.395090 18137 leader_election.cc:288] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819 [CANDIDATE]: Term 1 election: Requested vote from peers 7e959ec3b76e4e31850d8986edb31f10 (127.2.56.2:45547), 73379ec00f7842ada1c253cf0175e4f5 (127.2.56.3:41343)
I20260928 12:24:11.395205 18070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004e417071894df998cf5aaf10bf4c2f" candidate_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73379ec00f7842ada1c253cf0175e4f5"
I20260928 12:24:11.395264 18070 raft_consensus.cc:3037] T 004e417071894df998cf5aaf10bf4c2f P 73379ec00f7842ada1c253cf0175e4f5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:24:11.395323 17936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004e417071894df998cf5aaf10bf4c2f" candidate_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e959ec3b76e4e31850d8986edb31f10"
I20260928 12:24:11.395381 17936 raft_consensus.cc:3037] T 004e417071894df998cf5aaf10bf4c2f P 7e959ec3b76e4e31850d8986edb31f10 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:24:11.396096 18070 raft_consensus.cc:2454] T 004e417071894df998cf5aaf10bf4c2f P 73379ec00f7842ada1c253cf0175e4f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3acf94f9ac746f7b4d817b2eb4d2819 in term 1.
I20260928 12:24:11.396160 17936 raft_consensus.cc:2454] T 004e417071894df998cf5aaf10bf4c2f P 7e959ec3b76e4e31850d8986edb31f10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3acf94f9ac746f7b4d817b2eb4d2819 in term 1.
I20260928 12:24:11.396260 17736 leader_election.cc:302] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73379ec00f7842ada1c253cf0175e4f5, e3acf94f9ac746f7b4d817b2eb4d2819; no voters:
I20260928 12:24:11.396353 18137 raft_consensus.cc:2781] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:24:11.396551 18137 raft_consensus.cc:686] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819 [term 1 LEADER]: Becoming Leader. State: Replica: e3acf94f9ac746f7b4d817b2eb4d2819, State: Running, Role: LEADER
I20260928 12:24:11.396638 18137 consensus_queue.cc:237] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e959ec3b76e4e31850d8986edb31f10" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 45547 } } peers { permanent_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37109 } } peers { permanent_uuid: "73379ec00f7842ada1c253cf0175e4f5" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41343 } }
I20260928 12:24:11.397395 17657 catalog_manager.cc:5495] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3acf94f9ac746f7b4d817b2eb4d2819 (127.2.56.1). New cstate: current_term: 1 leader_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e959ec3b76e4e31850d8986edb31f10" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 45547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73379ec00f7842ada1c253cf0175e4f5" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41343 } health_report { overall_health: UNKNOWN } } }
I20260928 12:24:11.404423 18128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:11.402292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51148 (local address 127.2.56.1:37109)
0928 12:24:11.402346 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 12:24:11.402349 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:24:11.402357 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:24:11.402426 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:24:11.402427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:24:11.402444 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:24:11.402497 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:24:11.402608 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:11.403059 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:11.403771 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:11.403970 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:11.404020 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:11.404151 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:24:11.404239 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:24:11.404304 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 12:24:11.404339 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 12:24:11.404359 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 12:24:11.405925 18070 raft_consensus.cc:1261] T 004e417071894df998cf5aaf10bf4c2f P 73379ec00f7842ada1c253cf0175e4f5 [term 1 FOLLOWER]: Refusing update from remote peer e3acf94f9ac746f7b4d817b2eb4d2819: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:24:11.406243 18140 consensus_queue.cc:1035] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73379ec00f7842ada1c253cf0175e4f5" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:24:11.406931 17936 raft_consensus.cc:1261] T 004e417071894df998cf5aaf10bf4c2f P 7e959ec3b76e4e31850d8986edb31f10 [term 1 FOLLOWER]: Refusing update from remote peer e3acf94f9ac746f7b4d817b2eb4d2819: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:24:11.407182 18140 consensus_queue.cc:1035] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e959ec3b76e4e31850d8986edb31f10" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 45547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:24:11.408380 18143 mvcc.cc:205] Tried to move back new op lower bound from 7334290437756448768 to 7334290437721133056. Current Snapshot: MvccSnapshot[applied={T|T < 7334290437756448768}]
I20260928 12:24:11.410140 18146 mvcc.cc:205] Tried to move back new op lower bound from 7334290437756448768 to 7334290437721133056. Current Snapshot: MvccSnapshot[applied={T|T < 7334290437756448768}]
I20260928 12:24:11.410954 17657 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:24:11.411038 17657 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:24:11.411661 17657 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed: Sending DeleteTablet for 3 replicas of tablet 004e417071894df998cf5aaf10bf4c2f
I20260928 12:24:11.411940 17782 tablet_service.cc:1514] Processing DeleteTablet for tablet 004e417071894df998cf5aaf10bf4c2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:24:11 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:51138
I20260928 12:24:11.411943 17916 tablet_service.cc:1514] Processing DeleteTablet for tablet 004e417071894df998cf5aaf10bf4c2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:24:11 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:36838
I20260928 12:24:11.412065 18050 tablet_service.cc:1514] Processing DeleteTablet for tablet 004e417071894df998cf5aaf10bf4c2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:24:11 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:49220
I20260928 12:24:11.412251 18155 tablet_replica.cc:330] T 004e417071894df998cf5aaf10bf4c2f P 73379ec00f7842ada1c253cf0175e4f5: stopping tablet replica
I20260928 12:24:11.412271 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 17719
I20260928 12:24:11.412338 18155 raft_consensus.cc:2229] T 004e417071894df998cf5aaf10bf4c2f P 73379ec00f7842ada1c253cf0175e4f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:24:11.412426 18155 raft_consensus.cc:2258] T 004e417071894df998cf5aaf10bf4c2f P 73379ec00f7842ada1c253cf0175e4f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:24:11.412940 18155 ts_tablet_manager.cc:1905] T 004e417071894df998cf5aaf10bf4c2f P 73379ec00f7842ada1c253cf0175e4f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:24:11.414110 18154 tablet_replica.cc:330] T 004e417071894df998cf5aaf10bf4c2f P 7e959ec3b76e4e31850d8986edb31f10: stopping tablet replica
I20260928 12:24:11.414222 18154 raft_consensus.cc:2229] T 004e417071894df998cf5aaf10bf4c2f P 7e959ec3b76e4e31850d8986edb31f10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:24:11.414316 18154 raft_consensus.cc:2258] T 004e417071894df998cf5aaf10bf4c2f P 7e959ec3b76e4e31850d8986edb31f10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:24:11.414669 18154 ts_tablet_manager.cc:1905] T 004e417071894df998cf5aaf10bf4c2f P 7e959ec3b76e4e31850d8986edb31f10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:24:11.414978 18155 ts_tablet_manager.cc:1918] T 004e417071894df998cf5aaf10bf4c2f P 73379ec00f7842ada1c253cf0175e4f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:24:11.415048 18155 log.cc:1196] T 004e417071894df998cf5aaf10bf4c2f P 73379ec00f7842ada1c253cf0175e4f5: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/wal/wals/004e417071894df998cf5aaf10bf4c2f
I20260928 12:24:11.415290 18155 ts_tablet_manager.cc:1939] T 004e417071894df998cf5aaf10bf4c2f P 73379ec00f7842ada1c253cf0175e4f5: Deleting consensus metadata
I20260928 12:24:11.415671 17642 catalog_manager.cc:4841] TS 73379ec00f7842ada1c253cf0175e4f5 (127.2.56.3:41343): tablet 004e417071894df998cf5aaf10bf4c2f (table test-table [id=5675c966d6d1452ea0aa67141f9f35f9]) successfully deleted
I20260928 12:24:11.415951 18154 ts_tablet_manager.cc:1918] T 004e417071894df998cf5aaf10bf4c2f P 7e959ec3b76e4e31850d8986edb31f10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:24:11.416011 18154 log.cc:1196] T 004e417071894df998cf5aaf10bf4c2f P 7e959ec3b76e4e31850d8986edb31f10: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/wal/wals/004e417071894df998cf5aaf10bf4c2f
I20260928 12:24:11.416222 18154 ts_tablet_manager.cc:1939] T 004e417071894df998cf5aaf10bf4c2f P 7e959ec3b76e4e31850d8986edb31f10: Deleting consensus metadata
I20260928 12:24:11.416694 17644 catalog_manager.cc:4841] TS 7e959ec3b76e4e31850d8986edb31f10 (127.2.56.2:45547): tablet 004e417071894df998cf5aaf10bf4c2f (table test-table [id=5675c966d6d1452ea0aa67141f9f35f9]) successfully deleted
W20260928 12:24:11.417544 17641 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.56.1:37109 (error 108)
I20260928 12:24:11.417757 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 17853
I20260928 12:24:11.418020 18125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:11.417908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:37109 (local address 127.0.0.1:51154)
0928 12:24:11.417954 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:11.417988 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.1:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 12:24:11.418634 17641 catalog_manager.cc:4584] TS e3acf94f9ac746f7b4d817b2eb4d2819 (127.2.56.1:37109): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 004e417071894df998cf5aaf10bf4c2f: Network error: Client connection negotiation failed: client connection to 127.2.56.1:37109: connect: Connection refused (error 111)
I20260928 12:24:11.422384 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 17987
I20260928 12:24:11.426429 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 17625
I20260928 12:24:31.430776 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:45011
--webserver_interface=127.2.56.62
--webserver_port=36613
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:37705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:45011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:24:31.509735 18157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:24:31.509918 18157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:24:31.509938 18157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:24:31.511078 18157 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 12:24:31.511117 18157 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 12:24:31.511128 18157 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 12:24:31.511148 18157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:24:31.511173 18157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:24:31.511196 18157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:24:31.511216 18157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:24:31.511237 18157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:24:31.512694 18157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:37705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:45011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:45011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=36613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:24:31.512917 18157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:24:31.513110 18157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:24:31.521059 18163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:24:31.521085 18166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:24:31.521178 18164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:24:31.521296 18157 server_base.cc:1034] running on GCE node
I20260928 12:24:31.521510 18157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:24:31.521785 18157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:24:31.522910 18157 hybrid_clock.cc:648] HybridClock initialized: now 1790598271522902 us; error 26 us; skew 500 ppm
Sep 28 12:24:31 dist-test-slave-920p krb5kdc[17612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598271, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:24:31.525255 18157 init.cc:377] Logged in from keytab as kudu/127.2.56.62@KRBTEST.COM (short username kudu)
I20260928 12:24:31.525712 18157 webserver.cc:460] Webserver started at http://127.2.56.62:36613/ using document root <none> and password file <none>
I20260928 12:24:31.525928 18157 fs_manager.cc:359] Metadata directory not provided
I20260928 12:24:31.525982 18157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:24:31.527326 18157 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:24:31.527932 18173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:24:31.528070 18157 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:24:31.528154 18157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "520b64e7c6ac4a4a88448716abebfbed"
format_stamp: "Formatted at 2026-09-28 12:24:10 on dist-test-slave-920p"
server_key: "56b5f746a078f88383a08f96a19eeb25"
server_key_iv: "00ce8be1862b53c60c1280b42ee36e02"
server_key_version: "encryptionkey@0"
I20260928 12:24:31.528558 18157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:24:31.576287 18157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:24:31.578327 18157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:24:31.578477 18157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:24:31.582207 18157 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:45011
I20260928 12:24:31.582243 18225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:45011 every 8 connection(s)
I20260928 12:24:31.582607 18157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:24:31.584041 18226 sys_catalog.cc:263] Verifying existing consensus state
I20260928 12:24:31.584638 18226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed: Bootstrap starting.
I20260928 12:24:31.585801 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 18157
I20260928 12:24:31.586040 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.1:37109
--local_ip_for_outbound_sockets=127.2.56.1
--tserver_master_addrs=127.2.56.62:45011
--webserver_port=38051
--webserver_interface=127.2.56.1
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:37705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:24:31.592969 18226 log.cc:824] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed: Log is configured to *not* fsync() on all Append() calls
I20260928 12:24:31.596506 18226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:24:31.596827 18226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed: Bootstrap complete.
I20260928 12:24:31.599058 18226 raft_consensus.cc:348] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520b64e7c6ac4a4a88448716abebfbed" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 45011 } }
I20260928 12:24:31.599299 18226 raft_consensus.cc:729] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 520b64e7c6ac4a4a88448716abebfbed, State: Initialized, Role: FOLLOWER
I20260928 12:24:31.599453 18226 consensus_queue.cc:260] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520b64e7c6ac4a4a88448716abebfbed" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 45011 } }
I20260928 12:24:31.599526 18226 raft_consensus.cc:388] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:24:31.599562 18226 raft_consensus.cc:482] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:24:31.599609 18226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [term 1 FOLLOWER]: Advancing to term 2
I20260928 12:24:31.600630 18226 raft_consensus.cc:504] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520b64e7c6ac4a4a88448716abebfbed" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 45011 } }
I20260928 12:24:31.600718 18226 leader_election.cc:302] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 520b64e7c6ac4a4a88448716abebfbed; no voters:
I20260928 12:24:31.600890 18226 leader_election.cc:288] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 12:24:31.600950 18230 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [term 2 FOLLOWER]: Leader election won for term 2
I20260928 12:24:31.601069 18230 raft_consensus.cc:686] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [term 2 LEADER]: Becoming Leader. State: Replica: 520b64e7c6ac4a4a88448716abebfbed, State: Running, Role: LEADER
I20260928 12:24:31.601162 18226 sys_catalog.cc:564] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:24:31.601151 18230 consensus_queue.cc:237] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520b64e7c6ac4a4a88448716abebfbed" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 45011 } }
I20260928 12:24:31.601492 18230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 520b64e7c6ac4a4a88448716abebfbed. Latest consensus state: current_term: 2 leader_uuid: "520b64e7c6ac4a4a88448716abebfbed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520b64e7c6ac4a4a88448716abebfbed" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 45011 } } }
I20260928 12:24:31.601563 18230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [sys.catalog]: This master's current role is: LEADER
I20260928 12:24:31.601723 18232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "520b64e7c6ac4a4a88448716abebfbed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520b64e7c6ac4a4a88448716abebfbed" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 45011 } } }
I20260928 12:24:31.601858 18235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:24:31.601867 18232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [sys.catalog]: This master's current role is: LEADER
I20260928 12:24:31.602685 18235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:24:31.602803 18235 catalog_manager.cc:1259] Loaded cluster ID: 51b36946c0ec4f13a19d46635ddc523f
I20260928 12:24:31.602837 18235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:24:31.604099 18235 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:24:31.604280 18235 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed: Loaded TSK: 0
I20260928 12:24:31.613095 18235 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed: Generated new TSK 1
I20260928 12:24:31.613197 18235 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 12:24:31.668869 18228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:24:31.669085 18228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:24:31.669116 18228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:24:31.670369 18228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:24:31.670411 18228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:24:31.670447 18228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.1
I20260928 12:24:31.672134 18228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:37705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.1:37109
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.56.1
--webserver_port=38051
--tserver_master_addrs=127.2.56.62:45011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.1
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:24:31.672384 18228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:24:31.672601 18228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:24:31.677611 18256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:24:31.677686 18253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:24:31.677711 18254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:24:31.678164 18228 server_base.cc:1034] running on GCE node
I20260928 12:24:31.678364 18228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:24:31.678586 18228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:24:31.679738 18228 hybrid_clock.cc:648] HybridClock initialized: now 1790598271679728 us; error 32 us; skew 500 ppm
Sep 28 12:24:31 dist-test-slave-920p krb5kdc[17612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598271, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:24:31.682293 18228 init.cc:377] Logged in from keytab as kudu/127.2.56.1@KRBTEST.COM (short username kudu)
I20260928 12:24:31.682824 18228 webserver.cc:460] Webserver started at http://127.2.56.1:38051/ using document root <none> and password file <none>
I20260928 12:24:31.683070 18228 fs_manager.cc:359] Metadata directory not provided
I20260928 12:24:31.683152 18228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:24:31.684849 18228 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:24:31.685653 18263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:24:31.685824 18228 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 12:24:31.685901 18228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/wal
uuid: "e3acf94f9ac746f7b4d817b2eb4d2819"
format_stamp: "Formatted at 2026-09-28 12:24:10 on dist-test-slave-920p"
server_key: "65e27cb6ff43496afc107dbdd9dd08ef"
server_key_iv: "0434006bac5b175cf3fc786a33059582"
server_key_version: "encryptionkey@0"
I20260928 12:24:31.686295 18228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:24:31.715828 18228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:24:31.717916 18228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:24:31.718029 18228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:24:31.718299 18228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:24:31.718719 18270 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 12:24:31.720593 18228 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 12:24:31.720654 18228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 12:24:31.720688 18228 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 12:24:31.721385 18228 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 12:24:31.721421 18228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260928 12:24:31.721482 18270 tablet_bootstrap.cc:492] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819: Bootstrap starting.
I20260928 12:24:31.727916 18228 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.1:37109
I20260928 12:24:31.728263 18228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
I20260928 12:24:31.729472 18377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.1:37109 every 8 connection(s)
I20260928 12:24:31.730997 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 18228
I20260928 12:24:31.731271 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.2:45547
--local_ip_for_outbound_sockets=127.2.56.2
--tserver_master_addrs=127.2.56.62:45011
--webserver_port=39979
--webserver_interface=127.2.56.2
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:37705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:24:31.731918 18270 log.cc:824] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819: Log is configured to *not* fsync() on all Append() calls
I20260928 12:24:31.734313 18270 tablet_bootstrap.cc:492] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:24:31.734580 18270 tablet_bootstrap.cc:492] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819: Bootstrap complete.
I20260928 12:24:31.735435 18270 ts_tablet_manager.cc:1397] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.011s
I20260928 12:24:31.736726 18270 raft_consensus.cc:348] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e959ec3b76e4e31850d8986edb31f10" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 45547 } } peers { permanent_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37109 } } peers { permanent_uuid: "73379ec00f7842ada1c253cf0175e4f5" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41343 } }
I20260928 12:24:31.737020 18270 raft_consensus.cc:729] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3acf94f9ac746f7b4d817b2eb4d2819, State: Initialized, Role: FOLLOWER
I20260928 12:24:31.737170 18270 consensus_queue.cc:260] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e959ec3b76e4e31850d8986edb31f10" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 45547 } } peers { permanent_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37109 } } peers { permanent_uuid: "73379ec00f7842ada1c253cf0175e4f5" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41343 } }
I20260928 12:24:31.737458 18270 ts_tablet_manager.cc:1428] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
Sep 28 12:24:31 dist-test-slave-920p krb5kdc[17612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598271, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:24:31.744104 18380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:31.728961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.1:49577)
0928 12:24:31.733510 (+ 4549us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:31.733554 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 12:24:31.734462 (+ 908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:24:31.734968 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:24:31.734973 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:24:31.735332 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:24:31.735618 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:24:31.735625 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:24:31.737193 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:24:31.737194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:24:31.738097 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:24:31.738099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:24:31.738643 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:24:31.738644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:24:31.738767 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:31.739254 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:24:31.739262 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:24:31.740730 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:24:31.742724 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:24:31.742731 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:24:31.742733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:24:31.742938 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:24:31.743187 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:24:31.743188 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:24:31.743188 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:24:31.743285 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:24:31.743663 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:24:31.743667 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:24:31.743779 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 12:24:31.743856 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 12:24:31.743968 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4508,"thread_start_us":91,"threads_started":1}
I20260928 12:24:31.744237 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:31.729044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:49577 (local address 127.2.56.62:45011)
0928 12:24:31.729224 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 12:24:31.729233 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:24:31.733671 (+ 4438us) server_negotiation.cc:407] Connection header received
0928 12:24:31.734701 (+ 1030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:24:31.734708 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:24:31.734764 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:24:31.734905 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:24:31.735744 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:31.737066 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:31.738195 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:31.738555 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:31.738654 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:31.740833 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:24:31.740837 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:24:31.740838 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:24:31.740850 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:24:31.742632 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:24:31.743016 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:24:31.743018 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:24:31.743019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:24:31.743121 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:24:31.743345 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:24:31.743346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:24:31.743346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:24:31.743603 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:24:31.743678 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 12:24:31.743925 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 12:24:31.744056 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":72,"threads_started":1}
I20260928 12:24:31.744804 18378 heartbeater.cc:347] Connected to a master server at 127.2.56.62:45011
I20260928 12:24:31.744899 18378 heartbeater.cc:464] Registering TS with master...
I20260928 12:24:31.745064 18378 heartbeater.cc:511] Master 127.2.56.62:45011 requested a full tablet report, sending...
I20260928 12:24:31.745790 18190 ts_manager.cc:194] Registered new tserver with Master: e3acf94f9ac746f7b4d817b2eb4d2819 (127.2.56.1:37109)
I20260928 12:24:31.746891 18190 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.1@KRBTEST.COM'} at 127.2.56.1:49577
I20260928 12:24:31.747752 18378 heartbeater.cc:503] Master 127.2.56.62:45011 was elected leader, sending a full tablet report...
Sep 28 12:24:31 dist-test-slave-920p krb5kdc[17612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598271, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for kudu/127.2.56.1@KRBTEST.COM
I20260928 12:24:31.754016 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:31.746360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:37109 (local address 127.0.0.1:54668)
0928 12:24:31.746507 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:31.746542 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 12:24:31.746604 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:24:31.746852 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:24:31.746857 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:24:31.747157 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:24:31.747208 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:24:31.747210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:24:31.748304 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:24:31.748305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:24:31.749215 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:24:31.749218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:24:31.749683 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:24:31.749686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:24:31.749761 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:31.750146 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:24:31.750156 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:24:31.751459 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:24:31.752951 (+ 1492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:24:31.752958 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:24:31.752959 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:24:31.753138 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:24:31.753361 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:24:31.753362 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:24:31.753362 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:24:31.753417 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:24:31.753665 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:24:31.753671 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:24:31.753720 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 12:24:31.753796 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 12:24:31.753917 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260928 12:24:31.754016 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:31.746558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54668 (local address 127.2.56.1:37109)
0928 12:24:31.746704 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 12:24:31.746708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:24:31.746717 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:24:31.746748 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:24:31.746749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:24:31.746761 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:24:31.746833 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:24:31.747297 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:31.748224 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:31.749328 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:31.749615 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:31.749667 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:31.751543 (+ 1876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:24:31.751550 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:24:31.751550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:24:31.751557 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:24:31.752868 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:24:31.753215 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:24:31.753217 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:24:31.753218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:24:31.753302 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:24:31.753472 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:24:31.753472 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:24:31.753473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:24:31.753601 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:24:31.753667 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 12:24:31.753837 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 12:24:31.753922 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":75,"threads_started":1}
I20260928 12:24:31.754467 18312 tablet_service.cc:1514] Processing DeleteTablet for tablet 004e417071894df998cf5aaf10bf4c2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:24:11 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:54668
I20260928 12:24:31.755076 18387 tablet_replica.cc:330] stopping tablet replica
I20260928 12:24:31.755172 18387 raft_consensus.cc:2229] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:24:31.755237 18387 raft_consensus.cc:2258] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:24:31.755501 18387 ts_tablet_manager.cc:1905] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:24:31.757747 18387 ts_tablet_manager.cc:1918] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:24:31.757818 18387 log.cc:1196] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/wal/wals/004e417071894df998cf5aaf10bf4c2f
I20260928 12:24:31.757987 18387 ts_tablet_manager.cc:1939] T 004e417071894df998cf5aaf10bf4c2f P e3acf94f9ac746f7b4d817b2eb4d2819: Deleting consensus metadata
I20260928 12:24:31.758437 18174 catalog_manager.cc:4841] TS e3acf94f9ac746f7b4d817b2eb4d2819 (127.2.56.1:37109): tablet 004e417071894df998cf5aaf10bf4c2f (table test-table [id=5675c966d6d1452ea0aa67141f9f35f9]) successfully deleted
W20260928 12:24:31.815234 18383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:24:31.815438 18383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:24:31.815460 18383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:24:31.816602 18383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:24:31.816643 18383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:24:31.816668 18383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.2
I20260928 12:24:31.818152 18383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:37705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.2:45547
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.56.2
--webserver_port=39979
--tserver_master_addrs=127.2.56.62:45011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.2
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:24:31.818373 18383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:24:31.818584 18383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:24:31.823223 18392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:24:31.823374 18393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:24:31.823235 18395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:24:31.823446 18383 server_base.cc:1034] running on GCE node
I20260928 12:24:31.823617 18383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:24:31.823802 18383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:24:31.824939 18383 hybrid_clock.cc:648] HybridClock initialized: now 1790598271824930 us; error 26 us; skew 500 ppm
Sep 28 12:24:31 dist-test-slave-920p krb5kdc[17612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598271, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:24:31.827244 18383 init.cc:377] Logged in from keytab as kudu/127.2.56.2@KRBTEST.COM (short username kudu)
I20260928 12:24:31.827626 18383 webserver.cc:460] Webserver started at http://127.2.56.2:39979/ using document root <none> and password file <none>
I20260928 12:24:31.827797 18383 fs_manager.cc:359] Metadata directory not provided
I20260928 12:24:31.827845 18383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:24:31.829028 18383 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:24:31.829602 18402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:24:31.829738 18383 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:24:31.829811 18383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/wal
uuid: "7e959ec3b76e4e31850d8986edb31f10"
format_stamp: "Formatted at 2026-09-28 12:24:11 on dist-test-slave-920p"
server_key: "b30db4177d546d8e45c6c86e9c6f3fce"
server_key_iv: "ec216594c0f1bf5c511832a81f62cb9c"
server_key_version: "encryptionkey@0"
I20260928 12:24:31.830121 18383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:24:31.840786 18383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:24:31.842844 18383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:24:31.842962 18383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:24:31.843166 18383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:24:31.843458 18383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:24:31.843490 18383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:24:31.843518 18383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:24:31.843534 18383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:24:31.848932 18383 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.2:45547
I20260928 12:24:31.848973 18515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.2:45547 every 8 connection(s)
I20260928 12:24:31.849346 18383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
I20260928 12:24:31.856099 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 18383
Sep 28 12:24:31 dist-test-slave-920p krb5kdc[17612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598271, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:24:31.856405 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.3:41343
--local_ip_for_outbound_sockets=127.2.56.3
--tserver_master_addrs=127.2.56.62:45011
--webserver_port=43413
--webserver_interface=127.2.56.3
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:37705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:24:31.859505 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:31.850869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:52443 (local address 127.2.56.62:45011)
0928 12:24:31.850932 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 12:24:31.850934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:24:31.851157 (+ 223us) server_negotiation.cc:407] Connection header received
0928 12:24:31.851586 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:24:31.851587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:24:31.851602 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:24:31.851661 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:24:31.852607 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:31.853120 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:31.854256 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:31.854466 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:31.854524 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:31.856904 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:24:31.856905 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:24:31.856906 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:24:31.856913 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:24:31.858165 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:24:31.858620 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:24:31.858621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:24:31.858621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:24:31.858679 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:24:31.858891 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:24:31.858891 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:24:31.858892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:24:31.859014 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:24:31.859076 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 12:24:31.859319 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 12:24:31.859413 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 12:24:31.859573 18518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:31.850798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.2:52443)
0928 12:24:31.850964 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:31.851008 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 12:24:31.851500 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:24:31.851719 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:24:31.851722 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:24:31.852299 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:24:31.852495 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:24:31.852500 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:24:31.853328 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:24:31.853329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:24:31.854119 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:24:31.854122 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:24:31.854565 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:24:31.854566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:24:31.854685 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:31.855279 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:24:31.855287 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:24:31.856797 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:24:31.858269 (+ 1472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:24:31.858277 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:24:31.858279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:24:31.858543 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:24:31.858741 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:24:31.858741 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:24:31.858741 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:24:31.858834 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:24:31.859068 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:24:31.859071 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:24:31.859180 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 12:24:31.859280 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 12:24:31.859406 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":69,"threads_started":1}
I20260928 12:24:31.860040 18516 heartbeater.cc:347] Connected to a master server at 127.2.56.62:45011
I20260928 12:24:31.860226 18516 heartbeater.cc:464] Registering TS with master...
I20260928 12:24:31.860445 18516 heartbeater.cc:511] Master 127.2.56.62:45011 requested a full tablet report, sending...
I20260928 12:24:31.860826 18190 ts_manager.cc:194] Registered new tserver with Master: 7e959ec3b76e4e31850d8986edb31f10 (127.2.56.2:45547)
I20260928 12:24:31.861169 18190 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.2@KRBTEST.COM'} at 127.2.56.2:52443
W20260928 12:24:31.935770 18519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:24:31.935967 18519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:24:31.935998 18519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:24:31.937124 18519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:24:31.937182 18519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:24:31.937225 18519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.3
I20260928 12:24:31.938691 18519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:37705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.3:41343
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.56.3
--webserver_port=43413
--tserver_master_addrs=127.2.56.62:45011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.3
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:24:31.938915 18519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:24:31.939118 18519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:24:31.943835 18525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:24:31.943843 18527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:24:31.944020 18519 server_base.cc:1034] running on GCE node
W20260928 12:24:31.943966 18524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:24:31.944273 18519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:24:31.944515 18519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:24:31.945701 18519 hybrid_clock.cc:648] HybridClock initialized: now 1790598271945687 us; error 28 us; skew 500 ppm
Sep 28 12:24:31 dist-test-slave-920p krb5kdc[17612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598271, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:24:31.948163 18519 init.cc:377] Logged in from keytab as kudu/127.2.56.3@KRBTEST.COM (short username kudu)
I20260928 12:24:31.948580 18519 webserver.cc:460] Webserver started at http://127.2.56.3:43413/ using document root <none> and password file <none>
I20260928 12:24:31.948777 18519 fs_manager.cc:359] Metadata directory not provided
I20260928 12:24:31.948835 18519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:24:31.950137 18519 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:24:31.950690 18534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:24:31.950866 18519 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:24:31.950948 18519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/wal
uuid: "73379ec00f7842ada1c253cf0175e4f5"
format_stamp: "Formatted at 2026-09-28 12:24:11 on dist-test-slave-920p"
server_key: "239e88530eee59979c4d28502e69c083"
server_key_iv: "d262034a29f19ea725b0c006595fde87"
server_key_version: "encryptionkey@0"
I20260928 12:24:31.951301 18519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:24:31.972678 18519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:24:31.974606 18519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:24:31.974720 18519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:24:31.974900 18519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:24:31.975178 18519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:24:31.975215 18519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:24:31.975244 18519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:24:31.975262 18519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:24:31.980377 18519 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.3:41343
I20260928 12:24:31.980417 18647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.3:41343 every 8 connection(s)
I20260928 12:24:31.980716 18519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
I20260928 12:24:31.981562 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 18519
Sep 28 12:24:31 dist-test-slave-920p krb5kdc[17612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598271, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:24:31.988206 18652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:31.981955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39344 (local address 127.2.56.62:45011)
0928 12:24:31.982075 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 12:24:31.982077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:24:31.982167 (+ 90us) server_negotiation.cc:407] Connection header received
0928 12:24:31.982238 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:24:31.982239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:24:31.982250 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:24:31.982325 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:24:31.982910 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:31.983416 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:31.984169 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:31.984364 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:31.984423 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:31.985643 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:24:31.985644 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:24:31.985645 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:24:31.985652 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:24:31.987107 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:24:31.987471 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:24:31.987472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:24:31.987472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:24:31.987527 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:24:31.987690 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:24:31.987691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:24:31.987691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:24:31.987794 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:24:31.987949 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 12:24:31.987985 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 12:24:31.988106 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
I20260928 12:24:31.989938 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:31.981358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:32903 (local address 127.2.56.62:45011)
0928 12:24:31.981428 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 12:24:31.981430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:24:31.981693 (+ 263us) server_negotiation.cc:407] Connection header received
0928 12:24:31.982296 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:24:31.982297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:24:31.982315 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:24:31.982368 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:24:31.983237 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:31.983774 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:31.984744 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:31.984945 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:31.984997 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:31.987196 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:24:31.987197 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:24:31.987197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:24:31.987203 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:24:31.988558 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:24:31.988999 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:24:31.989001 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:24:31.989001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:24:31.989060 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:24:31.989344 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:24:31.989345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:24:31.989345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:24:31.989451 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:24:31.989504 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 12:24:31.989774 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 12:24:31.989858 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 12:24:31.990096 18650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:31.981371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.3:32903)
0928 12:24:31.981553 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:31.981599 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 12:24:31.982220 (+ 621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:24:31.982435 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:24:31.982439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:24:31.982967 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:24:31.983145 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:24:31.983149 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:24:31.983859 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:24:31.983860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:24:31.984661 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:24:31.984664 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:24:31.985017 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:24:31.985018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:24:31.985128 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:31.985528 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:24:31.985535 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:24:31.987121 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:24:31.988627 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:24:31.988633 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:24:31.988634 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:24:31.988932 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:24:31.989159 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:24:31.989160 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:24:31.989160 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:24:31.989289 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:24:31.989510 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:24:31.989516 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:24:31.989644 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 12:24:31.989724 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 12:24:31.989893 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":86,"threads_started":1}
I20260928 12:24:31.990486 18648 heartbeater.cc:347] Connected to a master server at 127.2.56.62:45011
I20260928 12:24:31.990567 18648 heartbeater.cc:464] Registering TS with master...
I20260928 12:24:31.990701 18648 heartbeater.cc:511] Master 127.2.56.62:45011 requested a full tablet report, sending...
I20260928 12:24:31.991008 18190 ts_manager.cc:194] Registered new tserver with Master: 73379ec00f7842ada1c253cf0175e4f5 (127.2.56.3:41343)
I20260928 12:24:31.991324 18190 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.3@KRBTEST.COM'} at 127.2.56.3:32903
I20260928 12:24:31.999894 2272 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:24:32.002621 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:32.000203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39356 (local address 127.2.56.62:45011)
0928 12:24:32.000262 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 12:24:32.000264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:24:32.000365 (+ 101us) server_negotiation.cc:407] Connection header received
0928 12:24:32.000418 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:24:32.000419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:24:32.000450 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:24:32.000518 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:24:32.000631 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:32.001141 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:32.001971 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:32.002150 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:32.002207 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:32.002401 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:24:32.002443 (+ 42us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 12:24:32.002561 (+ 118us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39356: token expired
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 12:24:32.002682 18381 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39356: token expired
W20260928 12:24:32.002780 18120 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.56.62:45011)
I20260928 12:24:32.008281 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:32.003001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39366 (local address 127.2.56.62:45011)
0928 12:24:32.003037 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 12:24:32.003039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:24:32.003046 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:24:32.003091 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:24:32.003091 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:24:32.003100 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:24:32.003146 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:24:32.003612 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:32.004059 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:32.004759 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:32.004941 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:32.004993 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:32.006152 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:24:32.006153 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:24:32.006154 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:24:32.006160 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:24:32.007296 (+ 1136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:24:32.007595 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:24:32.007596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:24:32.007596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:24:32.007650 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:24:32.007819 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:24:32.007820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:24:32.007820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:24:32.007923 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:24:32.008008 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 12:24:32.008107 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 12:24:32.008184 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 12:24:32.009611 18190 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39366:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:24:32.011266 18190 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:24:32.013341 18312 tablet_service.cc:1467] Processing CreateTablet for tablet f8cf5c4a5a884e0e80756899ed201b11 (DEFAULT_TABLE table=test-table [id=1925a317af96491381a0607e8d434a7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:24:32.013547 18312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8cf5c4a5a884e0e80756899ed201b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:24:32.015074 18270 tablet_bootstrap.cc:492] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819: Bootstrap starting.
I20260928 12:24:32.015494 18270 tablet_bootstrap.cc:654] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819: Neither blocks nor log segments found. Creating new log.
I20260928 12:24:32.016217 18270 tablet_bootstrap.cc:492] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819: No bootstrap required, opened a new log
I20260928 12:24:32.016265 18270 ts_tablet_manager.cc:1397] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:24:32.016371 18270 raft_consensus.cc:348] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73379ec00f7842ada1c253cf0175e4f5" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41343 } } peers { permanent_uuid: "7e959ec3b76e4e31850d8986edb31f10" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 45547 } } peers { permanent_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37109 } }
I20260928 12:24:32.016438 18270 raft_consensus.cc:374] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:24:32.016489 18270 raft_consensus.cc:729] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3acf94f9ac746f7b4d817b2eb4d2819, State: Initialized, Role: FOLLOWER
I20260928 12:24:32.016528 18270 consensus_queue.cc:260] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73379ec00f7842ada1c253cf0175e4f5" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41343 } } peers { permanent_uuid: "7e959ec3b76e4e31850d8986edb31f10" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 45547 } } peers { permanent_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37109 } }
I20260928 12:24:32.016556 18654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:32.012901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:45547 (local address 127.0.0.1:47900)
0928 12:24:32.013076 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:32.013086 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:24:32.013132 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:24:32.013590 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:24:32.013591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:24:32.013594 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:24:32.013640 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:24:32.013642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:24:32.014527 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:24:32.014528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:24:32.015405 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:24:32.015407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:24:32.016324 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:24:32.016324 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:24:32.016390 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:32.016411 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 12:24:32.016438 (+ 27us) client_negotiation.cc:240] Negotiation successful
0928 12:24:32.016478 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":11,"thread_start_us":112,"threads_started":1}
I20260928 12:24:32.016608 18270 ts_tablet_manager.cc:1428] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 12:24:32.016925 18656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:32.013164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47900 (local address 127.2.56.2:45547)
0928 12:24:32.013362 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 12:24:32.013366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:24:32.013375 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:24:32.013406 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:24:32.013407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:24:32.013420 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:24:32.013493 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:24:32.013696 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:32.014449 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:32.015499 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:32.016224 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:32.016707 (+ 483us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:32.016781 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 12:24:32.016808 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 12:24:32.016847 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":83,"threads_started":1}
I20260928 12:24:32.017040 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:32.012901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:41343 (local address 127.0.0.1:49026)
0928 12:24:32.012949 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:32.012959 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:24:32.012994 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:24:32.013567 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:24:32.013568 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:24:32.013570 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:24:32.013616 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:24:32.013618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:24:32.014932 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:24:32.014933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:24:32.015920 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:24:32.015923 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:24:32.016847 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:24:32.016849 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:24:32.016917 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:32.016921 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:24:32.016949 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 12:24:32.016985 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 12:24:32.017340 18655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:32.012994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49026 (local address 127.2.56.3:41343)
0928 12:24:32.013392 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0928 12:24:32.013395 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:24:32.013406 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:24:32.013442 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:24:32.013444 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:24:32.013458 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:24:32.013687 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:24:32.013776 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:32.014831 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:32.016070 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:32.016752 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:32.017121 (+ 369us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:32.017182 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 12:24:32.017209 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 12:24:32.017246 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":105,"threads_started":1}
I20260928 12:24:32.017607 18450 tablet_service.cc:1467] Processing CreateTablet for tablet f8cf5c4a5a884e0e80756899ed201b11 (DEFAULT_TABLE table=test-table [id=1925a317af96491381a0607e8d434a7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:24:32.017946 18450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8cf5c4a5a884e0e80756899ed201b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:24:32.017901 18582 tablet_service.cc:1467] Processing CreateTablet for tablet f8cf5c4a5a884e0e80756899ed201b11 (DEFAULT_TABLE table=test-table [id=1925a317af96491381a0607e8d434a7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:24:32.018232 18582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8cf5c4a5a884e0e80756899ed201b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:24:32.020704 18658 tablet_bootstrap.cc:492] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10: Bootstrap starting.
I20260928 12:24:32.020870 18659 tablet_bootstrap.cc:492] T f8cf5c4a5a884e0e80756899ed201b11 P 73379ec00f7842ada1c253cf0175e4f5: Bootstrap starting.
I20260928 12:24:32.021442 18659 tablet_bootstrap.cc:654] T f8cf5c4a5a884e0e80756899ed201b11 P 73379ec00f7842ada1c253cf0175e4f5: Neither blocks nor log segments found. Creating new log.
I20260928 12:24:32.021461 18658 tablet_bootstrap.cc:654] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10: Neither blocks nor log segments found. Creating new log.
I20260928 12:24:32.021891 18659 log.cc:824] T f8cf5c4a5a884e0e80756899ed201b11 P 73379ec00f7842ada1c253cf0175e4f5: Log is configured to *not* fsync() on all Append() calls
I20260928 12:24:32.021903 18658 log.cc:824] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10: Log is configured to *not* fsync() on all Append() calls
I20260928 12:24:32.022486 18659 tablet_bootstrap.cc:492] T f8cf5c4a5a884e0e80756899ed201b11 P 73379ec00f7842ada1c253cf0175e4f5: No bootstrap required, opened a new log
I20260928 12:24:32.022523 18658 tablet_bootstrap.cc:492] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10: No bootstrap required, opened a new log
I20260928 12:24:32.022538 18659 ts_tablet_manager.cc:1397] T f8cf5c4a5a884e0e80756899ed201b11 P 73379ec00f7842ada1c253cf0175e4f5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:24:32.022575 18658 ts_tablet_manager.cc:1397] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:24:32.023988 18659 raft_consensus.cc:348] T f8cf5c4a5a884e0e80756899ed201b11 P 73379ec00f7842ada1c253cf0175e4f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73379ec00f7842ada1c253cf0175e4f5" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41343 } } peers { permanent_uuid: "7e959ec3b76e4e31850d8986edb31f10" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 45547 } } peers { permanent_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37109 } }
I20260928 12:24:32.024119 18659 raft_consensus.cc:374] T f8cf5c4a5a884e0e80756899ed201b11 P 73379ec00f7842ada1c253cf0175e4f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:24:32.024142 18659 raft_consensus.cc:729] T f8cf5c4a5a884e0e80756899ed201b11 P 73379ec00f7842ada1c253cf0175e4f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73379ec00f7842ada1c253cf0175e4f5, State: Initialized, Role: FOLLOWER
I20260928 12:24:32.024031 18658 raft_consensus.cc:348] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73379ec00f7842ada1c253cf0175e4f5" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41343 } } peers { permanent_uuid: "7e959ec3b76e4e31850d8986edb31f10" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 45547 } } peers { permanent_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37109 } }
I20260928 12:24:32.024173 18658 raft_consensus.cc:374] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:24:32.024216 18658 raft_consensus.cc:729] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e959ec3b76e4e31850d8986edb31f10, State: Initialized, Role: FOLLOWER
I20260928 12:24:32.024251 18659 consensus_queue.cc:260] T f8cf5c4a5a884e0e80756899ed201b11 P 73379ec00f7842ada1c253cf0175e4f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73379ec00f7842ada1c253cf0175e4f5" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41343 } } peers { permanent_uuid: "7e959ec3b76e4e31850d8986edb31f10" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 45547 } } peers { permanent_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37109 } }
I20260928 12:24:32.024329 18658 consensus_queue.cc:260] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73379ec00f7842ada1c253cf0175e4f5" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41343 } } peers { permanent_uuid: "7e959ec3b76e4e31850d8986edb31f10" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 45547 } } peers { permanent_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37109 } }
I20260928 12:24:32.024507 18659 ts_tablet_manager.cc:1428] T f8cf5c4a5a884e0e80756899ed201b11 P 73379ec00f7842ada1c253cf0175e4f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:24:32.024554 18648 heartbeater.cc:503] Master 127.2.56.62:45011 was elected leader, sending a full tablet report...
I20260928 12:24:32.024626 18516 heartbeater.cc:503] Master 127.2.56.62:45011 was elected leader, sending a full tablet report...
I20260928 12:24:32.024847 18658 ts_tablet_manager.cc:1428] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260928 12:24:32.099946 18517 tablet.cc:2369] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:24:32.224222 18663 raft_consensus.cc:482] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:24:32.224418 18663 raft_consensus.cc:504] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73379ec00f7842ada1c253cf0175e4f5" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41343 } } peers { permanent_uuid: "7e959ec3b76e4e31850d8986edb31f10" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 45547 } } peers { permanent_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37109 } }
I20260928 12:24:32.224855 18663 leader_election.cc:288] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73379ec00f7842ada1c253cf0175e4f5 (127.2.56.3:41343), e3acf94f9ac746f7b4d817b2eb4d2819 (127.2.56.1:37109)
I20260928 12:24:32.227886 18655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:32.225293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:40383 (local address 127.2.56.3:41343)
0928 12:24:32.225348 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 12:24:32.225351 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:24:32.225393 (+ 42us) server_negotiation.cc:407] Connection header received
0928 12:24:32.225422 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:24:32.225423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:24:32.225434 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:24:32.225483 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:24:32.225688 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:32.226186 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:32.227170 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:32.227546 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:32.227605 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:32.227618 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 12:24:32.227771 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 12:24:32.227811 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 12:24:32.227886 18518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:32.225105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:41343 (local address 127.2.56.2:40383)
0928 12:24:32.225191 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:32.225208 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 12:24:32.225342 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:24:32.225523 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:24:32.225525 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:24:32.225529 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:24:32.225597 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:24:32.225599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:24:32.226274 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:24:32.226276 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:24:32.227073 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:24:32.227075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:24:32.227624 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:24:32.227625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:24:32.227701 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:32.227704 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:24:32.227750 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 12:24:32.227800 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 12:24:32.228356 18664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:32.225105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:37109 (local address 127.2.56.2:51747)
0928 12:24:32.225393 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:32.225403 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:24:32.225462 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:24:32.225634 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:24:32.225635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:24:32.225637 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:24:32.225728 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:24:32.225731 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:24:32.226467 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:24:32.226468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:24:32.227267 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:24:32.227269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:24:32.227908 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:24:32.227909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:24:32.228133 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:32.228139 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 12:24:32.228221 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 12:24:32.228267 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":54,"thread_start_us":129,"threads_started":1}
I20260928 12:24:32.228328 18602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cf5c4a5a884e0e80756899ed201b11" candidate_uuid: "7e959ec3b76e4e31850d8986edb31f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73379ec00f7842ada1c253cf0175e4f5" is_pre_election: true
I20260928 12:24:32.228490 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:32.225215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:51747 (local address 127.2.56.1:37109)
0928 12:24:32.225290 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 12:24:32.225292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:24:32.225475 (+ 183us) server_negotiation.cc:407] Connection header received
0928 12:24:32.225516 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:24:32.225518 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:24:32.225552 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:24:32.225613 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:24:32.225796 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:32.226375 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:32.227366 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:32.227818 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:32.227885 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:32.227991 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 12:24:32.228308 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 12:24:32.228411 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 12:24:32.228498 18602 raft_consensus.cc:2454] T f8cf5c4a5a884e0e80756899ed201b11 P 73379ec00f7842ada1c253cf0175e4f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e959ec3b76e4e31850d8986edb31f10 in term 0.
I20260928 12:24:32.228758 18404 leader_election.cc:302] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73379ec00f7842ada1c253cf0175e4f5, 7e959ec3b76e4e31850d8986edb31f10; no voters:
I20260928 12:24:32.228792 18332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cf5c4a5a884e0e80756899ed201b11" candidate_uuid: "7e959ec3b76e4e31850d8986edb31f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" is_pre_election: true
I20260928 12:24:32.228912 18663 raft_consensus.cc:2781] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:24:32.228971 18663 raft_consensus.cc:482] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:24:32.229001 18663 raft_consensus.cc:3037] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:24:32.228997 18332 raft_consensus.cc:2454] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e959ec3b76e4e31850d8986edb31f10 in term 0.
I20260928 12:24:32.229826 18663 raft_consensus.cc:504] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73379ec00f7842ada1c253cf0175e4f5" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41343 } } peers { permanent_uuid: "7e959ec3b76e4e31850d8986edb31f10" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 45547 } } peers { permanent_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37109 } }
I20260928 12:24:32.229976 18663 leader_election.cc:288] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10 [CANDIDATE]: Term 1 election: Requested vote from peers 73379ec00f7842ada1c253cf0175e4f5 (127.2.56.3:41343), e3acf94f9ac746f7b4d817b2eb4d2819 (127.2.56.1:37109)
I20260928 12:24:32.230146 18602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cf5c4a5a884e0e80756899ed201b11" candidate_uuid: "7e959ec3b76e4e31850d8986edb31f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73379ec00f7842ada1c253cf0175e4f5"
I20260928 12:24:32.230149 18332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cf5c4a5a884e0e80756899ed201b11" candidate_uuid: "7e959ec3b76e4e31850d8986edb31f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819"
I20260928 12:24:32.230216 18602 raft_consensus.cc:3037] T f8cf5c4a5a884e0e80756899ed201b11 P 73379ec00f7842ada1c253cf0175e4f5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:24:32.230237 18332 raft_consensus.cc:3037] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:24:32.230877 18602 raft_consensus.cc:2454] T f8cf5c4a5a884e0e80756899ed201b11 P 73379ec00f7842ada1c253cf0175e4f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e959ec3b76e4e31850d8986edb31f10 in term 1.
I20260928 12:24:32.231178 18404 leader_election.cc:302] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73379ec00f7842ada1c253cf0175e4f5, 7e959ec3b76e4e31850d8986edb31f10; no voters:
W20260928 12:24:32.231251 18649 tablet.cc:2369] T f8cf5c4a5a884e0e80756899ed201b11 P 73379ec00f7842ada1c253cf0175e4f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:24:32.231292 18663 raft_consensus.cc:2781] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:24:32.231441 18663 raft_consensus.cc:686] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10 [term 1 LEADER]: Becoming Leader. State: Replica: 7e959ec3b76e4e31850d8986edb31f10, State: Running, Role: LEADER
I20260928 12:24:32.231527 18663 consensus_queue.cc:237] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73379ec00f7842ada1c253cf0175e4f5" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41343 } } peers { permanent_uuid: "7e959ec3b76e4e31850d8986edb31f10" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 45547 } } peers { permanent_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37109 } }
I20260928 12:24:32.231715 18332 raft_consensus.cc:2454] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e959ec3b76e4e31850d8986edb31f10 in term 1.
I20260928 12:24:32.232497 18189 catalog_manager.cc:5495] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e959ec3b76e4e31850d8986edb31f10 (127.2.56.2). New cstate: current_term: 1 leader_uuid: "7e959ec3b76e4e31850d8986edb31f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73379ec00f7842ada1c253cf0175e4f5" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e959ec3b76e4e31850d8986edb31f10" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 45547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37109 } health_report { overall_health: UNKNOWN } } }
W20260928 12:24:32.233812 18379 tablet.cc:2369] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:24:32.246685 18656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:32.244233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47914 (local address 127.2.56.2:45547)
0928 12:24:32.244343 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 12:24:32.244346 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:24:32.244356 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:24:32.244400 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:24:32.244401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:24:32.244425 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:24:32.244529 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:24:32.244681 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:32.245188 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:32.245932 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:32.246141 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:32.246197 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:32.246321 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:24:32.246450 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:24:32.246513 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 12:24:32.246534 (+ 21us) server_negotiation.cc:299] Negotiation successful
0928 12:24:32.246552 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 12:24:32.248853 18602 raft_consensus.cc:1261] T f8cf5c4a5a884e0e80756899ed201b11 P 73379ec00f7842ada1c253cf0175e4f5 [term 1 FOLLOWER]: Refusing update from remote peer 7e959ec3b76e4e31850d8986edb31f10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:24:32.248903 18332 raft_consensus.cc:1261] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819 [term 1 FOLLOWER]: Refusing update from remote peer 7e959ec3b76e4e31850d8986edb31f10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:24:32.249352 18665 consensus_queue.cc:1035] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:24:32.249511 18663 consensus_queue.cc:1035] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73379ec00f7842ada1c253cf0175e4f5" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:24:32.252308 18668 mvcc.cc:205] Tried to move back new op lower bound from 7334290523128467456 to 7334290523060957184. Current Snapshot: MvccSnapshot[applied={T|T < 7334290523128467456}]
I20260928 12:24:32.252436 18669 mvcc.cc:205] Tried to move back new op lower bound from 7334290523128467456 to 7334290523060957184. Current Snapshot: MvccSnapshot[applied={T|T < 7334290523128467456}]
I20260928 12:24:32.254899 18189 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:24:32.255023 18189 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:24:32.255841 18189 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed: Sending DeleteTablet for 3 replicas of tablet f8cf5c4a5a884e0e80756899ed201b11
I20260928 12:24:32.256187 18582 tablet_service.cc:1514] Processing DeleteTablet for tablet f8cf5c4a5a884e0e80756899ed201b11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:24:32 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:49026
I20260928 12:24:32.256222 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 18228
I20260928 12:24:32.256292 18312 tablet_service.cc:1514] Processing DeleteTablet for tablet f8cf5c4a5a884e0e80756899ed201b11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:24:32 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:54668
I20260928 12:24:32.256453 18450 tablet_service.cc:1514] Processing DeleteTablet for tablet f8cf5c4a5a884e0e80756899ed201b11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:24:32 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:47900
I20260928 12:24:32.258527 18678 tablet_replica.cc:330] T f8cf5c4a5a884e0e80756899ed201b11 P 73379ec00f7842ada1c253cf0175e4f5: stopping tablet replica
I20260928 12:24:32.258663 18678 raft_consensus.cc:2229] T f8cf5c4a5a884e0e80756899ed201b11 P 73379ec00f7842ada1c253cf0175e4f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:24:32.258795 18678 raft_consensus.cc:2258] T f8cf5c4a5a884e0e80756899ed201b11 P 73379ec00f7842ada1c253cf0175e4f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:24:32.259333 18678 ts_tablet_manager.cc:1905] T f8cf5c4a5a884e0e80756899ed201b11 P 73379ec00f7842ada1c253cf0175e4f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:24:32.261412 18678 ts_tablet_manager.cc:1918] T f8cf5c4a5a884e0e80756899ed201b11 P 73379ec00f7842ada1c253cf0175e4f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:24:32.261497 18678 log.cc:1196] T f8cf5c4a5a884e0e80756899ed201b11 P 73379ec00f7842ada1c253cf0175e4f5: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/wal/wals/f8cf5c4a5a884e0e80756899ed201b11
I20260928 12:24:32.261788 18678 ts_tablet_manager.cc:1939] T f8cf5c4a5a884e0e80756899ed201b11 P 73379ec00f7842ada1c253cf0175e4f5: Deleting consensus metadata
I20260928 12:24:32.262419 18175 catalog_manager.cc:4841] TS 73379ec00f7842ada1c253cf0175e4f5 (127.2.56.3:41343): tablet f8cf5c4a5a884e0e80756899ed201b11 (table test-table [id=1925a317af96491381a0607e8d434a7f]) successfully deleted
W20260928 12:24:32.262598 18174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.56.1:37109 (error 108)
I20260928 12:24:32.262642 18679 tablet_replica.cc:330] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10: stopping tablet replica
I20260928 12:24:32.262804 18679 raft_consensus.cc:2229] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:24:32.262934 18679 raft_consensus.cc:2258] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:24:32.263095 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:32.262975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:37109 (local address 127.0.0.1:54678)
0928 12:24:32.263026 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:32.263061 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.1:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 12:24:32.263166 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 18383
W20260928 12:24:32.263228 18174 catalog_manager.cc:4584] TS e3acf94f9ac746f7b4d817b2eb4d2819 (127.2.56.1:37109): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f8cf5c4a5a884e0e80756899ed201b11: Network error: Client connection negotiation failed: client connection to 127.2.56.1:37109: connect: Connection refused (error 111)
I20260928 12:24:32.269169 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 18519
I20260928 12:24:32.269197 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:32.269090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:45547 (local address 127.0.0.1:47916)
0928 12:24:32.269140 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:32.269166 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.2:45547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 12:24:32.274326 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 18157
I20260928 12:24:52.279562 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:45011
--webserver_interface=127.2.56.62
--webserver_port=36613
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:37705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:45011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:24:52.357864 18681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:24:52.358049 18681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:24:52.358070 18681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:24:52.359159 18681 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 12:24:52.359207 18681 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 12:24:52.359220 18681 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 12:24:52.359229 18681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:24:52.359241 18681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:24:52.359253 18681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:24:52.359267 18681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:24:52.359295 18681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:24:52.360723 18681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:37705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:45011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:45011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=36613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:24:52.360940 18681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:24:52.361135 18681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:24:52.365864 18689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:24:52.365942 18687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:24:52.366075 18681 server_base.cc:1034] running on GCE node
W20260928 12:24:52.365947 18686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:24:52.366371 18681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:24:52.366652 18681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:24:52.367796 18681 hybrid_clock.cc:648] HybridClock initialized: now 1790598292367783 us; error 28 us; skew 500 ppm
Sep 28 12:24:52 dist-test-slave-920p krb5kdc[17612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598292, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:24:52.370190 18681 init.cc:377] Logged in from keytab as kudu/127.2.56.62@KRBTEST.COM (short username kudu)
I20260928 12:24:52.370592 18681 webserver.cc:460] Webserver started at http://127.2.56.62:36613/ using document root <none> and password file <none>
I20260928 12:24:52.370788 18681 fs_manager.cc:359] Metadata directory not provided
I20260928 12:24:52.370833 18681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:24:52.372146 18681 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:24:52.372778 18696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:24:52.372967 18681 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:24:52.373051 18681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "520b64e7c6ac4a4a88448716abebfbed"
format_stamp: "Formatted at 2026-09-28 12:24:10 on dist-test-slave-920p"
server_key: "56b5f746a078f88383a08f96a19eeb25"
server_key_iv: "00ce8be1862b53c60c1280b42ee36e02"
server_key_version: "encryptionkey@0"
I20260928 12:24:52.373421 18681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:24:52.392932 18681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:24:52.394928 18681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:24:52.395059 18681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:24:52.398676 18681 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:45011
I20260928 12:24:52.398721 18748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:45011 every 8 connection(s)
I20260928 12:24:52.399056 18681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:24:52.400472 18749 sys_catalog.cc:263] Verifying existing consensus state
I20260928 12:24:52.401087 18749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed: Bootstrap starting.
I20260928 12:24:52.404115 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 18681
I20260928 12:24:52.404349 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.1:37109
--local_ip_for_outbound_sockets=127.2.56.1
--tserver_master_addrs=127.2.56.62:45011
--webserver_port=38051
--webserver_interface=127.2.56.1
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:37705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:24:52.409222 18749 log.cc:824] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed: Log is configured to *not* fsync() on all Append() calls
I20260928 12:24:52.411320 18749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:24:52.411592 18749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed: Bootstrap complete.
I20260928 12:24:52.413687 18749 raft_consensus.cc:348] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520b64e7c6ac4a4a88448716abebfbed" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 45011 } }
I20260928 12:24:52.413902 18749 raft_consensus.cc:729] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 520b64e7c6ac4a4a88448716abebfbed, State: Initialized, Role: FOLLOWER
I20260928 12:24:52.414040 18749 consensus_queue.cc:260] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520b64e7c6ac4a4a88448716abebfbed" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 45011 } }
I20260928 12:24:52.414100 18749 raft_consensus.cc:388] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:24:52.414139 18749 raft_consensus.cc:482] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:24:52.414184 18749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [term 2 FOLLOWER]: Advancing to term 3
I20260928 12:24:52.415093 18749 raft_consensus.cc:504] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520b64e7c6ac4a4a88448716abebfbed" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 45011 } }
I20260928 12:24:52.415181 18749 leader_election.cc:302] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 520b64e7c6ac4a4a88448716abebfbed; no voters:
I20260928 12:24:52.415329 18749 leader_election.cc:288] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 12:24:52.415371 18753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [term 3 FOLLOWER]: Leader election won for term 3
I20260928 12:24:52.415486 18753 raft_consensus.cc:686] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [term 3 LEADER]: Becoming Leader. State: Replica: 520b64e7c6ac4a4a88448716abebfbed, State: Running, Role: LEADER
I20260928 12:24:52.415567 18753 consensus_queue.cc:237] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520b64e7c6ac4a4a88448716abebfbed" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 45011 } }
I20260928 12:24:52.415606 18749 sys_catalog.cc:564] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:24:52.415822 18753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 520b64e7c6ac4a4a88448716abebfbed. Latest consensus state: current_term: 3 leader_uuid: "520b64e7c6ac4a4a88448716abebfbed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520b64e7c6ac4a4a88448716abebfbed" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 45011 } } }
I20260928 12:24:52.415882 18753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [sys.catalog]: This master's current role is: LEADER
I20260928 12:24:52.415844 18754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "520b64e7c6ac4a4a88448716abebfbed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520b64e7c6ac4a4a88448716abebfbed" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 45011 } } }
I20260928 12:24:52.415944 18754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed [sys.catalog]: This master's current role is: LEADER
I20260928 12:24:52.416074 18757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:24:52.416879 18757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:24:52.416977 18757 catalog_manager.cc:1259] Loaded cluster ID: 51b36946c0ec4f13a19d46635ddc523f
I20260928 12:24:52.417006 18757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:24:52.418134 18757 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:24:52.418325 18757 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed: Loaded TSK: 0
I20260928 12:24:52.418371 18757 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 520b64e7c6ac4a4a88448716abebfbed: Loaded TSK: 1
F20260928 12:24:52.418578 18757 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd5838ab9ad google::LogMessage::Fail() at ??:0
@ 0x7fd5838ae88a google::LogMessage::SendToLog() at ??:0
@ 0x7fd5838ab4de google::LogMessage::Flush() at ??:0
@ 0x7fd5838aca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd586018104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fd586015d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fd58b8051cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fd58b80548e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fd58b805594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fd58b7cbc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fd58b7d5ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd5857790d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd58576e954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd5865d26db start_thread at ??:0
@ 0x7fd5823a961f clone at ??:0
W20260928 12:24:52.514278 18751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:24:52.514549 18751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:24:52.514585 18751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:24:52.516273 18751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:24:52.516326 18751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:24:52.516363 18751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.1
I20260928 12:24:52.518469 18751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:37705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.1:37109
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.56.1
--webserver_port=38051
--tserver_master_addrs=127.2.56.62:45011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.1
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:24:52.518719 18751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:24:52.518976 18751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:24:52.524072 18790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:24:52.524179 18751 server_base.cc:1034] running on GCE node
W20260928 12:24:52.524119 18791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:24:52.524289 18793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:24:52.524612 18751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:24:52.524889 18751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:24:52.526043 18751 hybrid_clock.cc:648] HybridClock initialized: now 1790598292526020 us; error 45 us; skew 500 ppm
Sep 28 12:24:52 dist-test-slave-920p krb5kdc[17612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598292, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:24:52.529306 18751 init.cc:377] Logged in from keytab as kudu/127.2.56.1@KRBTEST.COM (short username kudu)
I20260928 12:24:52.529865 18751 webserver.cc:460] Webserver started at http://127.2.56.1:38051/ using document root <none> and password file <none>
I20260928 12:24:52.530121 18751 fs_manager.cc:359] Metadata directory not provided
I20260928 12:24:52.530187 18751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:24:52.531905 18751 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:24:52.532642 18800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:24:52.532809 18751 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:24:52.532902 18751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/wal
uuid: "e3acf94f9ac746f7b4d817b2eb4d2819"
format_stamp: "Formatted at 2026-09-28 12:24:10 on dist-test-slave-920p"
server_key: "65e27cb6ff43496afc107dbdd9dd08ef"
server_key_iv: "0434006bac5b175cf3fc786a33059582"
server_key_version: "encryptionkey@0"
I20260928 12:24:52.533371 18751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:24:52.554798 18751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:24:52.557305 18751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:24:52.557461 18751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:24:52.557725 18751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:24:52.558240 18807 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 12:24:52.559689 18751 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 12:24:52.559746 18751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 12:24:52.559777 18751 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 12:24:52.560613 18751 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 12:24:52.560657 18751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 12:24:52.560724 18807 tablet_bootstrap.cc:492] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819: Bootstrap starting.
I20260928 12:24:52.567391 18751 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.1:37109
I20260928 12:24:52.567569 18914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.1:37109 every 8 connection(s)
I20260928 12:24:52.567829 18751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
I20260928 12:24:52.569286 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 18751
I20260928 12:24:52.569576 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.2:45547
--local_ip_for_outbound_sockets=127.2.56.2
--tserver_master_addrs=127.2.56.62:45011
--webserver_port=39979
--webserver_interface=127.2.56.2
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:37705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:24:52.570456 18807 log.cc:824] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819: Log is configured to *not* fsync() on all Append() calls
I20260928 12:24:52.572955 18807 tablet_bootstrap.cc:492] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:24:52.573251 18807 tablet_bootstrap.cc:492] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819: Bootstrap complete.
I20260928 12:24:52.574358 18807 ts_tablet_manager.cc:1397] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.010s
I20260928 12:24:52.575763 18807 raft_consensus.cc:348] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73379ec00f7842ada1c253cf0175e4f5" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41343 } } peers { permanent_uuid: "7e959ec3b76e4e31850d8986edb31f10" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 45547 } } peers { permanent_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37109 } }
I20260928 12:24:52.576099 18807 raft_consensus.cc:729] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3acf94f9ac746f7b4d817b2eb4d2819, State: Initialized, Role: FOLLOWER
I20260928 12:24:52.576242 18807 consensus_queue.cc:260] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73379ec00f7842ada1c253cf0175e4f5" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41343 } } peers { permanent_uuid: "7e959ec3b76e4e31850d8986edb31f10" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 45547 } } peers { permanent_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37109 } }
I20260928 12:24:52.576505 18807 ts_tablet_manager.cc:1428] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
W20260928 12:24:52.633002 18917 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:24:52.568545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.1:57831)
0928 12:24:52.568751 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:52.568814 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 12:24:52.569857 (+ 1043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:24:52.632776 (+ 62919us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":102,"threads_started":1}
W20260928 12:24:52.633291 18915 heartbeater.cc:644] Failed to heartbeat to 127.2.56.62:45011 (0 consecutive failures): Network error: Failed to ping master at 127.2.56.62:45011: Client connection negotiation failed: client connection to 127.2.56.62:45011: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 12:24:52.633620 18917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:52.633494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.1:39895)
0928 12:24:52.633549 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:52.633595 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 12:24:52.633915 18917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:52.633841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.1:46355)
0928 12:24:52.633878 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:52.633900 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 12:24:52.634037 18915 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 12:24:52.651665 18919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:24:52.651872 18919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:24:52.651894 18919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:24:52.652999 18919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:24:52.653034 18919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:24:52.653059 18919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.2
I20260928 12:24:52.654627 18919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:37705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.2:45547
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.56.2
--webserver_port=39979
--tserver_master_addrs=127.2.56.62:45011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.2
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:24:52.654812 18919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:24:52.654989 18919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:24:52.659742 18928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:24:52.659735 18925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:24:52.659758 18919 server_base.cc:1034] running on GCE node
W20260928 12:24:52.659754 18926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:24:52.660197 18919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:24:52.660398 18919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:24:52.661537 18919 hybrid_clock.cc:648] HybridClock initialized: now 1790598292661525 us; error 30 us; skew 500 ppm
Sep 28 12:24:52 dist-test-slave-920p krb5kdc[17612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598292, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:24:52.664047 18919 init.cc:377] Logged in from keytab as kudu/127.2.56.2@KRBTEST.COM (short username kudu)
I20260928 12:24:52.664534 18919 webserver.cc:460] Webserver started at http://127.2.56.2:39979/ using document root <none> and password file <none>
I20260928 12:24:52.664721 18919 fs_manager.cc:359] Metadata directory not provided
I20260928 12:24:52.664760 18919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:24:52.666175 18919 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:24:52.666800 18935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:24:52.666963 18919 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:24:52.667104 18919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/wal
uuid: "7e959ec3b76e4e31850d8986edb31f10"
format_stamp: "Formatted at 2026-09-28 12:24:11 on dist-test-slave-920p"
server_key: "b30db4177d546d8e45c6c86e9c6f3fce"
server_key_iv: "ec216594c0f1bf5c511832a81f62cb9c"
server_key_version: "encryptionkey@0"
I20260928 12:24:52.667461 18919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:24:52.691972 18919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:24:52.693862 18919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:24:52.693966 18919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:24:52.694146 18919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:24:52.694572 18942 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 12:24:52.695968 18919 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 12:24:52.696012 18919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 12:24:52.696036 18919 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 12:24:52.696643 18919 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 12:24:52.696674 18919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 12:24:52.696769 18942 tablet_bootstrap.cc:492] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10: Bootstrap starting.
I20260928 12:24:52.703474 18919 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.2:45547
I20260928 12:24:52.703642 19049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.2:45547 every 8 connection(s)
I20260928 12:24:52.703879 18919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
I20260928 12:24:52.705006 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:52.704510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.2:40259)
0928 12:24:52.704698 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:52.704800 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":69,"threads_started":1}
W20260928 12:24:52.705222 19050 heartbeater.cc:644] Failed to heartbeat to 127.2.56.62:45011 (0 consecutive failures): Network error: Failed to ping master at 127.2.56.62:45011: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
I20260928 12:24:52.705246 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 18919
I20260928 12:24:52.705529 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.3:41343
--local_ip_for_outbound_sockets=127.2.56.3
--tserver_master_addrs=127.2.56.62:45011
--webserver_port=43413
--webserver_interface=127.2.56.3
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:37705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:24:52.705845 19053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:52.705436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.2:55645)
0928 12:24:52.705763 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:52.705808 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20260928 12:24:52.706113 19053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:52.706022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.2:55207)
0928 12:24:52.706076 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:52.706094 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 12:24:52.706190 19050 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 12:24:52.707111 18942 log.cc:824] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10: Log is configured to *not* fsync() on all Append() calls
I20260928 12:24:52.709249 18942 tablet_bootstrap.cc:492] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:24:52.709555 18942 tablet_bootstrap.cc:492] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10: Bootstrap complete.
I20260928 12:24:52.710675 18942 ts_tablet_manager.cc:1397] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.010s
I20260928 12:24:52.712520 18942 raft_consensus.cc:348] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73379ec00f7842ada1c253cf0175e4f5" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41343 } } peers { permanent_uuid: "7e959ec3b76e4e31850d8986edb31f10" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 45547 } } peers { permanent_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37109 } }
I20260928 12:24:52.712893 18942 raft_consensus.cc:729] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e959ec3b76e4e31850d8986edb31f10, State: Initialized, Role: FOLLOWER
I20260928 12:24:52.713047 18942 consensus_queue.cc:260] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73379ec00f7842ada1c253cf0175e4f5" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41343 } } peers { permanent_uuid: "7e959ec3b76e4e31850d8986edb31f10" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 45547 } } peers { permanent_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37109 } }
I20260928 12:24:52.713374 18942 ts_tablet_manager.cc:1428] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260928 12:24:52.713653 19053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:52.713528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.2:45593)
0928 12:24:52.713587 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:52.713625 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 12:24:52.713945 19053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:52.713861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.2:60047)
0928 12:24:52.713902 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:52.713926 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 12:24:52.790427 19054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:24:52.790650 19054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:24:52.790673 19054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:24:52.791853 19054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:24:52.791898 19054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:24:52.791922 19054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.3
I20260928 12:24:52.793391 19054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:37705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.3:41343
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.56.3
--webserver_port=43413
--tserver_master_addrs=127.2.56.62:45011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.3
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:24:52.793619 19054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:24:52.793833 19054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:24:52.798544 19061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:24:52.798542 19064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:24:52.798738 19062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:24:52.798745 19054 server_base.cc:1034] running on GCE node
I20260928 12:24:52.798964 19054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:24:52.799198 19054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:24:52.800339 19054 hybrid_clock.cc:648] HybridClock initialized: now 1790598292800326 us; error 28 us; skew 500 ppm
Sep 28 12:24:52 dist-test-slave-920p krb5kdc[17612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598292, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:24:52.802678 19054 init.cc:377] Logged in from keytab as kudu/127.2.56.3@KRBTEST.COM (short username kudu)
I20260928 12:24:52.803102 19054 webserver.cc:460] Webserver started at http://127.2.56.3:43413/ using document root <none> and password file <none>
I20260928 12:24:52.803308 19054 fs_manager.cc:359] Metadata directory not provided
I20260928 12:24:52.803354 19054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:24:52.804564 19054 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:24:52.805094 19071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:24:52.805282 19054 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:24:52.805370 19054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/wal
uuid: "73379ec00f7842ada1c253cf0175e4f5"
format_stamp: "Formatted at 2026-09-28 12:24:11 on dist-test-slave-920p"
server_key: "239e88530eee59979c4d28502e69c083"
server_key_iv: "d262034a29f19ea725b0c006595fde87"
server_key_version: "encryptionkey@0"
I20260928 12:24:52.805689 19054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:24:52.815953 19054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:24:52.817911 19054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:24:52.818037 19054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:24:52.818234 19054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:24:52.818518 19054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:24:52.818562 19054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:24:52.818590 19054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:24:52.818612 19054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:24:52.823992 19054 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.3:41343
I20260928 12:24:52.824054 19184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.3:41343 every 8 connection(s)
I20260928 12:24:52.824347 19054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
I20260928 12:24:52.825292 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:52.824930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.3:56979)
0928 12:24:52.825063 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:52.825128 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":66,"threads_started":1}
W20260928 12:24:52.825642 19185 heartbeater.cc:644] Failed to heartbeat to 127.2.56.62:45011 (0 consecutive failures): Network error: Failed to ping master at 127.2.56.62:45011: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
I20260928 12:24:52.825860 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:52.825785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.3:43511)
0928 12:24:52.825819 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:52.825843 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 12:24:52.826148 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:52.826006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.3:42063)
0928 12:24:52.826095 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:52.826126 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 12:24:52.826254 19185 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 12:24:52.832468 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 19054
I20260928 12:24:53.634904 19189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:53.634547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.1:36369)
0928 12:24:53.634734 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:53.634808 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":76,"threads_started":1}
I20260928 12:24:53.714725 19190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:53.714360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.2:50351)
0928 12:24:53.714584 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:53.714657 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":88,"threads_started":1}
I20260928 12:24:53.826896 19191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:53.826601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.3:57365)
0928 12:24:53.826768 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:53.826825 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":67,"threads_started":1}
I20260928 12:24:53.885100 19192 raft_consensus.cc:482] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:24:53.885319 19192 raft_consensus.cc:504] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73379ec00f7842ada1c253cf0175e4f5" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41343 } } peers { permanent_uuid: "7e959ec3b76e4e31850d8986edb31f10" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 45547 } } peers { permanent_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37109 } }
I20260928 12:24:53.885666 19192 leader_election.cc:288] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73379ec00f7842ada1c253cf0175e4f5 (127.2.56.3:41343), 7e959ec3b76e4e31850d8986edb31f10 (127.2.56.2:45547)
Sep 28 12:24:53 dist-test-slave-920p krb5kdc[17612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598292, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for kudu/127.2.56.3@KRBTEST.COM
Sep 28 12:24:53 dist-test-slave-920p krb5kdc[17612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598292, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for kudu/127.2.56.2@KRBTEST.COM
I20260928 12:24:53.896905 19193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:53.885900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:41343 (local address 127.2.56.1:54675)
0928 12:24:53.886101 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:53.886109 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:24:53.886152 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:24:53.887412 (+ 1260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:24:53.887414 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:24:53.887632 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:24:53.887711 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:24:53.887714 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:24:53.889019 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:24:53.889020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:24:53.889804 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:24:53.889807 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:24:53.890425 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:24:53.890426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:24:53.890542 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:53.891060 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:24:53.891071 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:24:53.892626 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:24:53.895121 (+ 2495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:24:53.895130 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:24:53.895133 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:24:53.895428 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:24:53.895722 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:24:53.895724 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:24:53.895724 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:24:53.895859 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:24:53.896386 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:24:53.896387 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:24:53.896473 (+ 86us) client_negotiation.cc:769] Sending connection context
0928 12:24:53.896574 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 12:24:53.896797 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":92,"thread_start_us":66,"threads_started":1}
I20260928 12:24:53.896939 19195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:53.886037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:54675 (local address 127.2.56.3:41343)
0928 12:24:53.886273 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 12:24:53.886286 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 12:24:53.886321 (+ 35us) server_negotiation.cc:407] Connection header received
0928 12:24:53.887182 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:24:53.887186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:24:53.887242 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:24:53.887406 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:24:53.887844 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:53.888897 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:53.889896 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:53.890319 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:53.890445 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:53.893077 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:24:53.893084 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:24:53.893085 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:24:53.893098 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:24:53.895019 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:24:53.895516 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:24:53.895521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:24:53.895522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:24:53.895636 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:24:53.895961 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:24:53.895962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:24:53.895962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:24:53.896301 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:24:53.896397 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 12:24:53.896653 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 12:24:53.896797 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":88,"threads_started":1}
I20260928 12:24:53.897022 19189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:53.885900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:45547 (local address 127.2.56.1:56825)
0928 12:24:53.886018 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:53.886033 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 12:24:53.886117 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:24:53.886622 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:24:53.886626 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:24:53.887105 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:24:53.887439 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:24:53.887450 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:24:53.888741 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:24:53.888742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:24:53.889802 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:24:53.889806 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:24:53.890413 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:24:53.890415 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:24:53.890545 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:53.891190 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:24:53.891195 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:24:53.892987 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:24:53.895160 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:24:53.895161 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:24:53.895161 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:24:53.895460 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:24:53.895811 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:24:53.895812 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:24:53.895813 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:24:53.895883 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:24:53.896288 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:24:53.896308 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:24:53.896475 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 12:24:53.896576 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 12:24:53.896886 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":139}
I20260928 12:24:53.897467 19194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:53.886030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:56825 (local address 127.2.56.2:45547)
0928 12:24:53.886244 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 12:24:53.886253 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:24:53.886277 (+ 24us) server_negotiation.cc:407] Connection header received
0928 12:24:53.886384 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:24:53.886386 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:24:53.886426 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:24:53.886574 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:24:53.887568 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:53.888575 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:53.889893 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:53.890289 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:53.890379 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:53.893112 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:24:53.893117 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:24:53.893117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:24:53.893127 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:24:53.895045 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:24:53.895581 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:24:53.895584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:24:53.895584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:24:53.895687 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:24:53.895964 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:24:53.895965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:24:53.895966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:24:53.896213 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:24:53.896283 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 12:24:53.896648 (+ 365us) server_negotiation.cc:299] Negotiation successful
0928 12:24:53.896780 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":93,"threads_started":1}
W20260928 12:24:53.897993 18802 leader_election.cc:341] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 73379ec00f7842ada1c253cf0175e4f5 (127.2.56.3:41343): Not found: Tablet not found: f8cf5c4a5a884e0e80756899ed201b11
I20260928 12:24:53.897977 19004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cf5c4a5a884e0e80756899ed201b11" candidate_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "7e959ec3b76e4e31850d8986edb31f10" is_pre_election: true
I20260928 12:24:53.898118 19004 raft_consensus.cc:2454] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3acf94f9ac746f7b4d817b2eb4d2819 in term 1.
I20260928 12:24:53.898361 18804 leader_election.cc:302] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7e959ec3b76e4e31850d8986edb31f10, e3acf94f9ac746f7b4d817b2eb4d2819; no voters: 73379ec00f7842ada1c253cf0175e4f5
I20260928 12:24:53.898587 19192 raft_consensus.cc:2781] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 12:24:53.898662 19192 raft_consensus.cc:482] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:24:53.898679 19192 raft_consensus.cc:3037] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819 [term 1 FOLLOWER]: Advancing to term 2
I20260928 12:24:53.900050 19192 raft_consensus.cc:504] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73379ec00f7842ada1c253cf0175e4f5" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41343 } } peers { permanent_uuid: "7e959ec3b76e4e31850d8986edb31f10" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 45547 } } peers { permanent_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37109 } }
I20260928 12:24:53.900209 19192 leader_election.cc:288] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819 [CANDIDATE]: Term 2 election: Requested vote from peers 73379ec00f7842ada1c253cf0175e4f5 (127.2.56.3:41343), 7e959ec3b76e4e31850d8986edb31f10 (127.2.56.2:45547)
I20260928 12:24:53.900450 19004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cf5c4a5a884e0e80756899ed201b11" candidate_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "7e959ec3b76e4e31850d8986edb31f10"
I20260928 12:24:53.900564 19004 raft_consensus.cc:3037] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10 [term 1 FOLLOWER]: Advancing to term 2
W20260928 12:24:53.900596 18802 leader_election.cc:341] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 73379ec00f7842ada1c253cf0175e4f5 (127.2.56.3:41343): Not found: Tablet not found: f8cf5c4a5a884e0e80756899ed201b11
I20260928 12:24:53.902021 19004 raft_consensus.cc:2454] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3acf94f9ac746f7b4d817b2eb4d2819 in term 2.
I20260928 12:24:53.902200 18804 leader_election.cc:302] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7e959ec3b76e4e31850d8986edb31f10, e3acf94f9ac746f7b4d817b2eb4d2819; no voters: 73379ec00f7842ada1c253cf0175e4f5
I20260928 12:24:53.902318 19192 raft_consensus.cc:2781] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 12:24:53.902460 19192 raft_consensus.cc:686] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819 [term 2 LEADER]: Becoming Leader. State: Replica: e3acf94f9ac746f7b4d817b2eb4d2819, State: Running, Role: LEADER
I20260928 12:24:53.902540 19192 consensus_queue.cc:237] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73379ec00f7842ada1c253cf0175e4f5" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41343 } } peers { permanent_uuid: "7e959ec3b76e4e31850d8986edb31f10" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 45547 } } peers { permanent_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37109 } }
I20260928 12:24:53.902946 19189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:53.902809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.1:47593)
0928 12:24:53.902876 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:53.902920 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 12:24:54.327018 18802 consensus_peers.cc:487] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819 -> Peer 73379ec00f7842ada1c253cf0175e4f5 (127.2.56.3:41343): Couldn't send request to peer 73379ec00f7842ada1c253cf0175e4f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f8cf5c4a5a884e0e80756899ed201b11. This is attempt 1: this message will repeat every 5th retry.
I20260928 12:24:54.522505 19004 raft_consensus.cc:1261] T f8cf5c4a5a884e0e80756899ed201b11 P 7e959ec3b76e4e31850d8986edb31f10 [term 2 FOLLOWER]: Refusing update from remote peer e3acf94f9ac746f7b4d817b2eb4d2819: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260928 12:24:54.522982 19192 consensus_queue.cc:1035] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e959ec3b76e4e31850d8986edb31f10" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 45547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260928 12:24:54.524129 19200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:54.523921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.2:41757)
0928 12:24:54.524037 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:54.524087 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":64,"threads_started":1}
I20260928 12:24:54.525079 19204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:54.524845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.1:58905)
0928 12:24:54.525013 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:54.525043 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":55,"threads_started":1}
I20260928 12:24:54.525306 19200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:54.525123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.2:35083)
0928 12:24:54.525228 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:54.525260 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260928 12:24:54.768352 19205 ts_tablet_manager.cc:927] T f8cf5c4a5a884e0e80756899ed201b11 P 73379ec00f7842ada1c253cf0175e4f5: Initiating tablet copy from peer e3acf94f9ac746f7b4d817b2eb4d2819 (127.2.56.1:37109)
I20260928 12:24:54.768594 19205 tablet_copy_client.cc:316] T f8cf5c4a5a884e0e80756899ed201b11 P 73379ec00f7842ada1c253cf0175e4f5: tablet copy: Beginning tablet copy session from remote peer at address 127.2.56.1:37109
Sep 28 12:24:54 dist-test-slave-920p krb5kdc[17612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598292, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for kudu/127.2.56.1@KRBTEST.COM
I20260928 12:24:54.776309 19208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:54.768989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:40205 (local address 127.2.56.1:37109)
0928 12:24:54.769217 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 12:24:54.769221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:24:54.769230 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:24:54.769253 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:24:54.769255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:24:54.769299 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:24:54.769370 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:24:54.769877 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:54.770595 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:54.771379 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:24:54.771670 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:24:54.771727 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:54.773857 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:24:54.773865 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:24:54.773866 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:24:54.773873 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:24:54.775221 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:24:54.775556 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:24:54.775558 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:24:54.775559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:24:54.775607 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:24:54.775772 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:24:54.775773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:24:54.775773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:24:54.775900 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:24:54.775968 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 12:24:54.776137 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 12:24:54.776208 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":117,"threads_started":1}
I20260928 12:24:54.776325 19207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:54.768868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:37109 (local address 127.2.56.3:40205)
0928 12:24:54.769008 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:54.769023 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 12:24:54.769077 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:24:54.769385 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:24:54.769392 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:24:54.769700 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:24:54.769796 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:24:54.769798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:24:54.770670 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:24:54.770671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:24:54.771302 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:24:54.771304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:24:54.771735 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:24:54.771736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:24:54.771815 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:24:54.772379 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:24:54.772385 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:24:54.773775 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:24:54.775300 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:24:54.775306 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:24:54.775308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:24:54.775482 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:24:54.775662 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:24:54.775662 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:24:54.775663 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:24:54.775715 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:24:54.775959 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:24:54.775965 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:24:54.776017 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 12:24:54.776092 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 12:24:54.776208 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20260928 12:24:54.776886 18889 tablet_copy_service.cc:140] P e3acf94f9ac746f7b4d817b2eb4d2819: Received BeginTabletCopySession request for tablet f8cf5c4a5a884e0e80756899ed201b11 from peer 73379ec00f7842ada1c253cf0175e4f5 ({username='kudu', principal='kudu/127.2.56.3@KRBTEST.COM'} at 127.2.56.3:40205)
I20260928 12:24:54.776952 18889 tablet_copy_service.cc:161] P e3acf94f9ac746f7b4d817b2eb4d2819: Beginning new tablet copy session on tablet f8cf5c4a5a884e0e80756899ed201b11 from peer 73379ec00f7842ada1c253cf0175e4f5 at {username='kudu', principal='kudu/127.2.56.3@KRBTEST.COM'} at 127.2.56.3:40205: session id = 73379ec00f7842ada1c253cf0175e4f5-f8cf5c4a5a884e0e80756899ed201b11
I20260928 12:24:54.777489 18889 tablet_copy_source_session.cc:205] T f8cf5c4a5a884e0e80756899ed201b11 P e3acf94f9ac746f7b4d817b2eb4d2819: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 12:24:54.778405 19205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8cf5c4a5a884e0e80756899ed201b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:24:54.780889 19205 tablet_copy_client.cc:791] T f8cf5c4a5a884e0e80756899ed201b11 P 73379ec00f7842ada1c253cf0175e4f5: tablet copy: Starting download of 0 data blocks...
I20260928 12:24:54.781030 19205 tablet_copy_client.cc:655] T f8cf5c4a5a884e0e80756899ed201b11 P 73379ec00f7842ada1c253cf0175e4f5: tablet copy: Starting download of 1 WAL segments...
I20260928 12:24:54.782080 19205 tablet_copy_client.cc:531] T f8cf5c4a5a884e0e80756899ed201b11 P 73379ec00f7842ada1c253cf0175e4f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 12:24:54.782805 19205 tablet_bootstrap.cc:492] T f8cf5c4a5a884e0e80756899ed201b11 P 73379ec00f7842ada1c253cf0175e4f5: Bootstrap starting.
I20260928 12:24:54.784104 19205 log.cc:824] T f8cf5c4a5a884e0e80756899ed201b11 P 73379ec00f7842ada1c253cf0175e4f5: Log is configured to *not* fsync() on all Append() calls
I20260928 12:24:54.786641 19205 tablet_bootstrap.cc:492] T f8cf5c4a5a884e0e80756899ed201b11 P 73379ec00f7842ada1c253cf0175e4f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:24:54.786849 19205 tablet_bootstrap.cc:492] T f8cf5c4a5a884e0e80756899ed201b11 P 73379ec00f7842ada1c253cf0175e4f5: Bootstrap complete.
I20260928 12:24:54.786969 19205 ts_tablet_manager.cc:1397] T f8cf5c4a5a884e0e80756899ed201b11 P 73379ec00f7842ada1c253cf0175e4f5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 12:24:54.788204 19205 raft_consensus.cc:348] T f8cf5c4a5a884e0e80756899ed201b11 P 73379ec00f7842ada1c253cf0175e4f5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73379ec00f7842ada1c253cf0175e4f5" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41343 } } peers { permanent_uuid: "7e959ec3b76e4e31850d8986edb31f10" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 45547 } } peers { permanent_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37109 } }
I20260928 12:24:54.788363 19205 raft_consensus.cc:729] T f8cf5c4a5a884e0e80756899ed201b11 P 73379ec00f7842ada1c253cf0175e4f5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73379ec00f7842ada1c253cf0175e4f5, State: Initialized, Role: FOLLOWER
I20260928 12:24:54.788496 19205 consensus_queue.cc:260] T f8cf5c4a5a884e0e80756899ed201b11 P 73379ec00f7842ada1c253cf0175e4f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73379ec00f7842ada1c253cf0175e4f5" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 41343 } } peers { permanent_uuid: "7e959ec3b76e4e31850d8986edb31f10" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 45547 } } peers { permanent_uuid: "e3acf94f9ac746f7b4d817b2eb4d2819" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37109 } }
I20260928 12:24:54.788770 19205 ts_tablet_manager.cc:1428] T f8cf5c4a5a884e0e80756899ed201b11 P 73379ec00f7842ada1c253cf0175e4f5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 12:24:54.789188 18889 tablet_copy_service.cc:342] P e3acf94f9ac746f7b4d817b2eb4d2819: Request end of tablet copy session 73379ec00f7842ada1c253cf0175e4f5-f8cf5c4a5a884e0e80756899ed201b11 received from {username='kudu', principal='kudu/127.2.56.3@KRBTEST.COM'} at 127.2.56.3:40205
I20260928 12:24:54.789248 18889 tablet_copy_service.cc:433] P e3acf94f9ac746f7b4d817b2eb4d2819: ending tablet copy session 73379ec00f7842ada1c253cf0175e4f5-f8cf5c4a5a884e0e80756899ed201b11 on tablet f8cf5c4a5a884e0e80756899ed201b11 with peer 73379ec00f7842ada1c253cf0175e4f5
I20260928 12:24:54.789245 19207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:54.789095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.3:56957)
0928 12:24:54.789168 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:54.789202 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 12:24:54.789633 19207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:54.789486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.3:57373)
0928 12:24:54.789554 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:54.789596 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 12:24:55.229650 19139 raft_consensus.cc:1203] T f8cf5c4a5a884e0e80756899ed201b11 P 73379ec00f7842ada1c253cf0175e4f5 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260928 12:24:55.229763 19212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:55.229492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.1:45533)
0928 12:24:55.229659 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:55.229713 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":81,"threads_started":1}
I20260928 12:24:55.230180 19207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:55.230064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.3:56137)
0928 12:24:55.230122 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:55.230150 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 12:24:55.526072 19213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:55.525740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.2:55729)
0928 12:24:55.525918 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:55.525988 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":55,"threads_started":1}
I20260928 12:24:56.230662 19214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:56.230337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.1:52455)
0928 12:24:56.230523 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:56.230597 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":100,"threads_started":1}
I20260928 12:24:56.231045 19215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:56.230791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.3:57669)
0928 12:24:56.230925 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:56.230972 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
I20260928 12:24:56.526875 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:56.526624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.2:58647)
0928 12:24:56.526780 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:56.526824 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":81,"threads_started":1}
I20260928 12:24:57.231524 19217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:57.231227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.1:42981)
0928 12:24:57.231416 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:57.231473 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":98,"threads_started":1}
I20260928 12:24:57.231724 19218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:57.231491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.3:42027)
0928 12:24:57.231613 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:57.231652 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":57,"threads_started":1}
I20260928 12:24:57.527675 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:57.527315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.2:45391)
0928 12:24:57.527576 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:57.527621 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":179,"threads_started":1}
I20260928 12:24:58.232470 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:58.232190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.1:55943)
0928 12:24:58.232341 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:58.232405 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260928 12:24:58.232579 19221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:58.232317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.3:59005)
0928 12:24:58.232480 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:58.232519 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":71,"threads_started":1}
I20260928 12:24:58.232901 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:58.232815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.1:57925)
0928 12:24:58.232857 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:58.232882 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 12:24:58.528543 19222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:58.528270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.2:46027)
0928 12:24:58.528441 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:58.528493 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":73,"threads_started":1}
I20260928 12:24:59.233572 19223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:59.233247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.3:52365)
0928 12:24:59.233462 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:59.233522 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":137,"threads_started":1}
I20260928 12:24:59.233637 19224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:59.233391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.1:36893)
0928 12:24:59.233529 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:59.233574 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":66,"threads_started":1}
I20260928 12:24:59.529434 19225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:24:59.529105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.2:41989)
0928 12:24:59.529322 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 12:24:59.529381 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":89,"threads_started":1}
I20260928 12:25:00.234442 19226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:00.234112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.3:58553)
0928 12:25:00.234329 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:00.234392 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":110,"threads_started":1}
I20260928 12:25:00.234428 19227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:00.234163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.1:58483)
0928 12:25:00.234317 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:00.234360 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":60,"threads_started":1}
I20260928 12:25:00.530268 19228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:00.529983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.2:57137)
0928 12:25:00.530157 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:00.530215 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":88,"threads_started":1}
I20260928 12:25:01.235337 19230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:01.235034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.1:35313)
0928 12:25:01.235228 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:01.235284 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":100,"threads_started":1}
I20260928 12:25:01.235520 19229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:01.235042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.3:37087)
0928 12:25:01.235260 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:01.235301 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":107,"threads_started":1}
I20260928 12:25:01.531140 19231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:01.530787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.2:60233)
0928 12:25:01.530985 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:01.531057 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":97,"threads_started":1}
I20260928 12:25:02.236325 19233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:02.236076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.3:51317)
0928 12:25:02.236236 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:02.236272 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":56,"threads_started":1}
I20260928 12:25:02.236397 19232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:02.236007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.1:35789)
0928 12:25:02.236242 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:02.236314 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":125,"threads_started":1}
I20260928 12:25:02.532240 19234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:02.531896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.2:44007)
0928 12:25:02.532130 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:02.532180 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":125,"threads_started":1}
I20260928 12:25:03.237198 19235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:03.236960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.3:53839)
0928 12:25:03.237105 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:03.237152 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":97,"threads_started":1}
I20260928 12:25:03.237248 19236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:03.236995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.1:58575)
0928 12:25:03.237145 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:03.237188 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":73,"threads_started":1}
I20260928 12:25:03.533094 19237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:03.532808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.2:53311)
0928 12:25:03.532970 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:03.533029 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":73,"threads_started":1}
I20260928 12:25:04.238097 19239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:04.237823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.1:49153)
0928 12:25:04.237987 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:04.238035 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":96,"threads_started":1}
I20260928 12:25:04.238130 19238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:04.237768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.3:55519)
0928 12:25:04.237984 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:04.238033 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":106,"threads_started":1}
I20260928 12:25:04.533843 19240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:04.533576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.2:60289)
0928 12:25:04.533740 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:04.533780 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":81,"threads_started":1}
I20260928 12:25:05.238775 19241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:05.238551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.1:33823)
0928 12:25:05.238688 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:05.238725 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":63,"threads_started":1}
I20260928 12:25:05.238893 19242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:05.238636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.3:36949)
0928 12:25:05.238785 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:05.238827 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":71,"threads_started":1}
I20260928 12:25:05.534552 19244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:05.534310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.2:42131)
0928 12:25:05.534446 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:05.534489 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":66,"threads_started":1}
I20260928 12:25:06.239634 19245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:06.239360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.1:44237)
0928 12:25:06.239533 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:06.239576 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":98,"threads_started":1}
I20260928 12:25:06.239653 19246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:06.239388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.3:45345)
0928 12:25:06.239551 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:06.239593 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":83,"threads_started":1}
I20260928 12:25:06.535579 19247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:06.535221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.2:41039)
0928 12:25:06.535458 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:06.535520 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":114,"threads_started":1}
I20260928 12:25:07.240406 19248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:07.240115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.1:60785)
0928 12:25:07.240286 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:07.240329 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":81,"threads_started":1}
I20260928 12:25:07.240408 19249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:07.240184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.3:52111)
0928 12:25:07.240313 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:07.240349 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":68,"threads_started":1}
I20260928 12:25:07.536552 19250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:07.536219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:45011 (local address 127.2.56.2:52315)
0928 12:25:07.536422 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:07.536487 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.62:45011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":77,"threads_started":1}
I20260928 12:25:07.836258 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 18751
I20260928 12:25:07.841033 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 18919
I20260928 12:25:07.845440 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 19054
2026-09-28T12:25:07Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58218 ms)
[----------] 30 tests from SecurityITest (235211 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:25:07 dist-test-slave-920p krb5kdc[19253](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:25:07 dist-test-slave-920p krb5kdc[19253](info): set up 2 sockets
Sep 28 12:25:07 dist-test-slave-920p krb5kdc[19253](info): commencing operation
krb5kdc: starting...
W20260928 12:25:08.820866 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:25:08 dist-test-slave-920p krb5kdc[19253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:25:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:25:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
I20260928 12:25:08.932202 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:33135
--webserver_interface=127.2.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:38483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:33135
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:25:09.012928 19266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:09.013120 19266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:09.013149 19266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:09.014300 19266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:25:09.014339 19266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:09.014358 19266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:09.014374 19266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:25:09.014390 19266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:25:09.015815 19266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:38483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:33135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:33135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:09.016043 19266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:09.016245 19266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:25:09.020967 19266 server_base.cc:1034] running on GCE node
W20260928 12:25:09.021028 19271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:09.021132 19272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:09.021028 19274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:09.021523 19266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:09.021792 19266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:09.022950 19266 hybrid_clock.cc:648] HybridClock initialized: now 1790598309022935 us; error 25 us; skew 500 ppm
Sep 28 12:25:09 dist-test-slave-920p krb5kdc[19253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598309, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:09.025177 19266 init.cc:377] Logged in from keytab as kudu/127.2.56.62@KRBTEST.COM (short username kudu)
I20260928 12:25:09.025588 19266 webserver.cc:460] Webserver started at http://127.2.56.62:43851/ using document root <none> and password file <none>
I20260928 12:25:09.025753 19266 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:09.025789 19266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:09.025871 19266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:09.026881 19266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "cae95ebd8ef344d28b0f545d0ce6895b"
format_stamp: "Formatted at 2026-09-28 12:25:09 on dist-test-slave-920p"
server_key: "3158e8ed84fe0f8c22a9e5277173fb55"
server_key_iv: "dacc54914d1406ed91625c97e50a2345"
server_key_version: "encryptionkey@0"
I20260928 12:25:09.027164 19266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "cae95ebd8ef344d28b0f545d0ce6895b"
format_stamp: "Formatted at 2026-09-28 12:25:09 on dist-test-slave-920p"
server_key: "3158e8ed84fe0f8c22a9e5277173fb55"
server_key_iv: "dacc54914d1406ed91625c97e50a2345"
server_key_version: "encryptionkey@0"
I20260928 12:25:09.028352 19266 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 12:25:09.029058 19281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:09.029244 19266 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:25:09.029342 19266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "cae95ebd8ef344d28b0f545d0ce6895b"
format_stamp: "Formatted at 2026-09-28 12:25:09 on dist-test-slave-920p"
server_key: "3158e8ed84fe0f8c22a9e5277173fb55"
server_key_iv: "dacc54914d1406ed91625c97e50a2345"
server_key_version: "encryptionkey@0"
I20260928 12:25:09.029399 19266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:09.046788 19266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:09.048419 19266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:09.048523 19266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:09.052026 19266 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:33135
I20260928 12:25:09.052070 19333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:33135 every 8 connection(s)
I20260928 12:25:09.052366 19266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:25:09.053172 19334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:09.055352 19334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cae95ebd8ef344d28b0f545d0ce6895b: Bootstrap starting.
I20260928 12:25:09.055881 19334 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cae95ebd8ef344d28b0f545d0ce6895b: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:09.056159 19334 log.cc:824] T 00000000000000000000000000000000 P cae95ebd8ef344d28b0f545d0ce6895b: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:09.056469 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 19266
I20260928 12:25:09.056563 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:25:09.056756 2272 external_mini_cluster.cc:1428] Setting key 1b72c2c7aed425a60883cf0d5b59d17f
I20260928 12:25:09.056879 19334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cae95ebd8ef344d28b0f545d0ce6895b: No bootstrap required, opened a new log
I20260928 12:25:09.059007 19334 raft_consensus.cc:348] T 00000000000000000000000000000000 P cae95ebd8ef344d28b0f545d0ce6895b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae95ebd8ef344d28b0f545d0ce6895b" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 33135 } }
I20260928 12:25:09.059162 19334 raft_consensus.cc:374] T 00000000000000000000000000000000 P cae95ebd8ef344d28b0f545d0ce6895b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:09.059186 19334 raft_consensus.cc:729] T 00000000000000000000000000000000 P cae95ebd8ef344d28b0f545d0ce6895b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae95ebd8ef344d28b0f545d0ce6895b, State: Initialized, Role: FOLLOWER
I20260928 12:25:09.059293 19334 consensus_queue.cc:260] T 00000000000000000000000000000000 P cae95ebd8ef344d28b0f545d0ce6895b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae95ebd8ef344d28b0f545d0ce6895b" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 33135 } }
I20260928 12:25:09.059361 19334 raft_consensus.cc:388] T 00000000000000000000000000000000 P cae95ebd8ef344d28b0f545d0ce6895b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:25:09.059384 19334 raft_consensus.cc:482] T 00000000000000000000000000000000 P cae95ebd8ef344d28b0f545d0ce6895b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:25:09.059428 19334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cae95ebd8ef344d28b0f545d0ce6895b [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:09.059932 19334 raft_consensus.cc:504] T 00000000000000000000000000000000 P cae95ebd8ef344d28b0f545d0ce6895b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae95ebd8ef344d28b0f545d0ce6895b" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 33135 } }
I20260928 12:25:09.060017 19334 leader_election.cc:302] T 00000000000000000000000000000000 P cae95ebd8ef344d28b0f545d0ce6895b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cae95ebd8ef344d28b0f545d0ce6895b; no voters:
I20260928 12:25:09.060184 19334 leader_election.cc:288] T 00000000000000000000000000000000 P cae95ebd8ef344d28b0f545d0ce6895b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:25:09.060241 19339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cae95ebd8ef344d28b0f545d0ce6895b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:25:09.060431 19339 raft_consensus.cc:686] T 00000000000000000000000000000000 P cae95ebd8ef344d28b0f545d0ce6895b [term 1 LEADER]: Becoming Leader. State: Replica: cae95ebd8ef344d28b0f545d0ce6895b, State: Running, Role: LEADER
I20260928 12:25:09.060499 19334 sys_catalog.cc:564] T 00000000000000000000000000000000 P cae95ebd8ef344d28b0f545d0ce6895b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:25:09.060529 19339 consensus_queue.cc:237] T 00000000000000000000000000000000 P cae95ebd8ef344d28b0f545d0ce6895b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae95ebd8ef344d28b0f545d0ce6895b" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 33135 } }
I20260928 12:25:09.060883 19339 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae95ebd8ef344d28b0f545d0ce6895b [sys.catalog]: SysCatalogTable state changed. Reason: New leader cae95ebd8ef344d28b0f545d0ce6895b. Latest consensus state: current_term: 1 leader_uuid: "cae95ebd8ef344d28b0f545d0ce6895b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae95ebd8ef344d28b0f545d0ce6895b" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 33135 } } }
I20260928 12:25:09.060952 19339 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae95ebd8ef344d28b0f545d0ce6895b [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:09.060943 19340 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae95ebd8ef344d28b0f545d0ce6895b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cae95ebd8ef344d28b0f545d0ce6895b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae95ebd8ef344d28b0f545d0ce6895b" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 33135 } } }
I20260928 12:25:09.061026 19340 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae95ebd8ef344d28b0f545d0ce6895b [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:09.061385 19346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:25:09.061847 19346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:25:09.063516 19346 catalog_manager.cc:1347] Generated new cluster ID: 675f4aa9e0774091b595edccef16d4d5
I20260928 12:25:09.063568 19346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:25:09 dist-test-slave-920p krb5kdc[19253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:25:09.067418 19338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:09.057349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51136 (local address 127.2.56.62:33135)
0928 12:25:09.057497 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:09.057505 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:09.057528 (+ 23us) server_negotiation.cc:407] Connection header received
0928 12:25:09.058134 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:09.058140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:09.058184 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:09.058306 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:09.059049 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.060443 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.061529 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.062091 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.062187 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:09.064571 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:09.064574 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:09.064575 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:09.064586 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:09.066077 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:09.066424 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:09.066426 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:09.066427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:09.066500 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:09.066669 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:09.066670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:09.066670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:09.066884 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:09.067022 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:09.067113 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 12:25:09.067235 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":72,"threads_started":1}
I20260928 12:25:09.094962 19346 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:25:09.095494 19346 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:25:09.114666 19346 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cae95ebd8ef344d28b0f545d0ce6895b: Generated new TSK 0
I20260928 12:25:09.114880 19346 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab.
Entry for principal kudu/127.2.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab.
I20260928 12:25:09.151635 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.1:0
--local_ip_for_outbound_sockets=127.2.56.1
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:33135
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:38483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:25:09.252444 19360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:09.252633 19360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:09.252655 19360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:09.253750 19360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:09.253786 19360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:09.253810 19360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.1
I20260928 12:25:09.255169 19360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:38483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:33135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.1
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:09.255344 19360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:09.255515 19360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:09.260002 19366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:09.260010 19368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:09.260017 19365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:09.260080 19360 server_base.cc:1034] running on GCE node
I20260928 12:25:09.260382 19360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:09.260679 19360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:09.261807 19360 hybrid_clock.cc:648] HybridClock initialized: now 1790598309261795 us; error 27 us; skew 500 ppm
Sep 28 12:25:09 dist-test-slave-920p krb5kdc[19253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598309, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:09.264053 19360 init.cc:377] Logged in from keytab as kudu/127.2.56.1@KRBTEST.COM (short username kudu)
I20260928 12:25:09.264427 19360 webserver.cc:460] Webserver started at http://127.2.56.1:39105/ using document root <none> and password file <none>
I20260928 12:25:09.264603 19360 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:09.264639 19360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:09.264724 19360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:09.265678 19360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-0/data/instance:
uuid: "8ea665cf0fb54026920d1e33b5c985b0"
format_stamp: "Formatted at 2026-09-28 12:25:09 on dist-test-slave-920p"
server_key: "c7b2fd11acb44d747917f93cfadfbd9a"
server_key_iv: "8324eb0c490918cf68446c52832c9949"
server_key_version: "encryptionkey@0"
I20260928 12:25:09.265944 19360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance:
uuid: "8ea665cf0fb54026920d1e33b5c985b0"
format_stamp: "Formatted at 2026-09-28 12:25:09 on dist-test-slave-920p"
server_key: "c7b2fd11acb44d747917f93cfadfbd9a"
server_key_iv: "8324eb0c490918cf68446c52832c9949"
server_key_version: "encryptionkey@0"
I20260928 12:25:09.267064 19360 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:25:09.267654 19375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:09.267813 19360 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:25:09.267884 19360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-0/wal
uuid: "8ea665cf0fb54026920d1e33b5c985b0"
format_stamp: "Formatted at 2026-09-28 12:25:09 on dist-test-slave-920p"
server_key: "c7b2fd11acb44d747917f93cfadfbd9a"
server_key_iv: "8324eb0c490918cf68446c52832c9949"
server_key_version: "encryptionkey@0"
I20260928 12:25:09.267935 19360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:09.309993 19360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:09.311841 19360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:09.311945 19360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:09.312144 19360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:25:09.312439 19360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:25:09.312471 19360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:09.312492 19360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:25:09.312515 19360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:09.318416 19360 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.1:45651
I20260928 12:25:09.318485 19488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.1:45651 every 8 connection(s)
I20260928 12:25:09.318749 19360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
Sep 28 12:25:09 dist-test-slave-920p krb5kdc[19253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598309, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:25:09.326794 19338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:09.319408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:51891 (local address 127.2.56.62:33135)
0928 12:25:09.319468 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:09.319470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:09.319651 (+ 181us) server_negotiation.cc:407] Connection header received
0928 12:25:09.320139 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:09.320140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:09.320151 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:09.320204 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:09.321068 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.321575 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.322509 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.322707 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.322763 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:09.324629 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:09.324630 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:09.324631 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:09.324637 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:09.325758 (+ 1121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:09.326097 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:09.326097 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:09.326098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:09.326145 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:09.326320 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:09.326321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:09.326321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:09.326422 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:09.326470 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:09.326665 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 12:25:09.326728 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 12:25:09.326946 19491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:09.319386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:33135 (local address 127.2.56.1:51891)
0928 12:25:09.319538 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:09.319577 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:09.320032 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:09.320263 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:09.320266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:09.320811 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:09.320996 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:09.321000 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.321644 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.321645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:09.322447 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:09.322449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.322772 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.322772 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:09.322866 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:09.323284 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:25:09.323291 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:09.324535 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:09.325829 (+ 1294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:09.325834 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:09.325836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:09.326044 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:09.326188 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:09.326189 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:09.326189 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:09.326276 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:09.326465 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:09.326467 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:09.326570 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 12:25:09.326645 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 12:25:09.326762 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":95,"threads_started":1}
I20260928 12:25:09.327421 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 19360
I20260928 12:25:09.327431 19489 heartbeater.cc:347] Connected to a master server at 127.2.56.62:33135
I20260928 12:25:09.327508 19489 heartbeater.cc:464] Registering TS with master...
I20260928 12:25:09.327517 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance
I20260928 12:25:09.327688 19489 heartbeater.cc:511] Master 127.2.56.62:33135 requested a full tablet report, sending...
I20260928 12:25:09.327698 2272 external_mini_cluster.cc:1428] Setting key ed98d73b869e675e533dd316d0f597b0
I20260928 12:25:09.328204 19297 ts_manager.cc:194] Registered new tserver with Master: 8ea665cf0fb54026920d1e33b5c985b0 (127.2.56.1:45651)
I20260928 12:25:09.328804 19297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.1@KRBTEST.COM'} at 127.2.56.1:51891
WARNING: no policy specified for kudu/127.2.56.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab.
Entry for principal kudu/127.2.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab.
I20260928 12:25:09.352604 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.2:0
--local_ip_for_outbound_sockets=127.2.56.2
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:33135
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:38483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:25:09.433065 19494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:09.433259 19494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:09.433305 19494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:09.434434 19494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:09.434475 19494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:09.434509 19494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.2
I20260928 12:25:09.435937 19494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:38483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:33135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.2
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:09.436158 19494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:09.436354 19494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:09.440953 19500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:09.440961 19499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:09.440961 19502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:09.441236 19494 server_base.cc:1034] running on GCE node
I20260928 12:25:09.441455 19494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:09.441677 19494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:09.442818 19494 hybrid_clock.cc:648] HybridClock initialized: now 1790598309442807 us; error 35 us; skew 500 ppm
Sep 28 12:25:09 dist-test-slave-920p krb5kdc[19253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598309, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:09.445076 19494 init.cc:377] Logged in from keytab as kudu/127.2.56.2@KRBTEST.COM (short username kudu)
I20260928 12:25:09.445498 19494 webserver.cc:460] Webserver started at http://127.2.56.2:37509/ using document root <none> and password file <none>
I20260928 12:25:09.445708 19494 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:09.445753 19494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:09.445873 19494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:09.446929 19494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-1/data/instance:
uuid: "969ac3ba7cd9488d922ae46380f7bc05"
format_stamp: "Formatted at 2026-09-28 12:25:09 on dist-test-slave-920p"
server_key: "7af7d3d1c500fc2f36a0a2eaf66b9a25"
server_key_iv: "4f66b5bacb14b62cb9abfdb47756cd78"
server_key_version: "encryptionkey@0"
I20260928 12:25:09.447247 19494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance:
uuid: "969ac3ba7cd9488d922ae46380f7bc05"
format_stamp: "Formatted at 2026-09-28 12:25:09 on dist-test-slave-920p"
server_key: "7af7d3d1c500fc2f36a0a2eaf66b9a25"
server_key_iv: "4f66b5bacb14b62cb9abfdb47756cd78"
server_key_version: "encryptionkey@0"
I20260928 12:25:09.448513 19494 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:25:09.449188 19509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:09.449358 19494 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:25:09.449436 19494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-1/wal
uuid: "969ac3ba7cd9488d922ae46380f7bc05"
format_stamp: "Formatted at 2026-09-28 12:25:09 on dist-test-slave-920p"
server_key: "7af7d3d1c500fc2f36a0a2eaf66b9a25"
server_key_iv: "4f66b5bacb14b62cb9abfdb47756cd78"
server_key_version: "encryptionkey@0"
I20260928 12:25:09.449503 19494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:09.459419 19494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:09.462827 19494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:09.462958 19494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:09.463191 19494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:25:09.463497 19494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:25:09.463537 19494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:09.463567 19494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:25:09.463583 19494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:09.469164 19494 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.2:38855
I20260928 12:25:09.469357 19622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.2:38855 every 8 connection(s)
I20260928 12:25:09.469548 19494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
Sep 28 12:25:09 dist-test-slave-920p krb5kdc[19253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598309, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:25:09.477084 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 19494
I20260928 12:25:09.477180 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance
I20260928 12:25:09.477399 2272 external_mini_cluster.cc:1428] Setting key 50ddf9fbef2ad6051c8a88c0dc41b00f
I20260928 12:25:09.478035 19338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:09.470210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:55255 (local address 127.2.56.62:33135)
0928 12:25:09.470256 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:09.470258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:09.470477 (+ 219us) server_negotiation.cc:407] Connection header received
0928 12:25:09.470990 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:09.470992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:09.471004 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:09.471088 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:09.472050 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.472546 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.473544 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.473746 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.473801 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:09.475704 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:09.475705 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:09.475706 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:09.475712 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:09.476723 (+ 1011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:09.477094 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:09.477095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:09.477096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:09.477148 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:09.477396 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:09.477397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:09.477397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:09.477524 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:09.477590 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:09.477872 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 12:25:09.477951 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:25:09.478127 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:09.470155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:33135 (local address 127.2.56.2:55255)
0928 12:25:09.470351 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:09.470396 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:09.470895 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:09.471158 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:09.471163 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:09.471760 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:09.471950 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:09.471955 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.472626 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.472627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:09.473465 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:09.473468 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.473813 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.473813 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:09.473933 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:09.474361 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:25:09.474369 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:09.475621 (+ 1252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:09.476793 (+ 1172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:09.476799 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:09.476801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:09.477026 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:09.477200 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:09.477201 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:09.477201 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:09.477339 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:09.477592 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:09.477598 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:09.477745 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 12:25:09.477837 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 12:25:09.477956 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":86,"threads_started":1}
I20260928 12:25:09.478552 19623 heartbeater.cc:347] Connected to a master server at 127.2.56.62:33135
I20260928 12:25:09.478641 19623 heartbeater.cc:464] Registering TS with master...
I20260928 12:25:09.478801 19623 heartbeater.cc:511] Master 127.2.56.62:33135 requested a full tablet report, sending...
I20260928 12:25:09.479152 19297 ts_manager.cc:194] Registered new tserver with Master: 969ac3ba7cd9488d922ae46380f7bc05 (127.2.56.2:38855)
I20260928 12:25:09.479579 19297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.2@KRBTEST.COM'} at 127.2.56.2:55255
WARNING: no policy specified for kudu/127.2.56.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab.
Entry for principal kudu/127.2.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab.
I20260928 12:25:09.500998 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.3:0
--local_ip_for_outbound_sockets=127.2.56.3
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:33135
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:38483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:25:09.579941 19628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:09.580142 19628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:09.580164 19628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:09.581239 19628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:09.581298 19628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:09.581341 19628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.3
I20260928 12:25:09.582722 19628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:38483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:33135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.3
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:09.582893 19628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:09.583076 19628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:09.587780 19634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:09.587781 19633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:09.587780 19636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:09.588012 19628 server_base.cc:1034] running on GCE node
I20260928 12:25:09.588203 19628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:09.588413 19628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:09.589553 19628 hybrid_clock.cc:648] HybridClock initialized: now 1790598309589537 us; error 26 us; skew 500 ppm
Sep 28 12:25:09 dist-test-slave-920p krb5kdc[19253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598309, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:09.592003 19628 init.cc:377] Logged in from keytab as kudu/127.2.56.3@KRBTEST.COM (short username kudu)
I20260928 12:25:09.592450 19628 webserver.cc:460] Webserver started at http://127.2.56.3:41877/ using document root <none> and password file <none>
I20260928 12:25:09.592649 19628 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:09.592715 19628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:09.592844 19628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:09.593928 19628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-2/data/instance:
uuid: "c36b7a7e5b7e48169411997e381066e9"
format_stamp: "Formatted at 2026-09-28 12:25:09 on dist-test-slave-920p"
server_key: "6fc877936d848fe3a1a2f3af47d62a98"
server_key_iv: "0cea13d3f4df28c9de2a5a297a1e3c1e"
server_key_version: "encryptionkey@0"
I20260928 12:25:09.594404 19628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance:
uuid: "c36b7a7e5b7e48169411997e381066e9"
format_stamp: "Formatted at 2026-09-28 12:25:09 on dist-test-slave-920p"
server_key: "6fc877936d848fe3a1a2f3af47d62a98"
server_key_iv: "0cea13d3f4df28c9de2a5a297a1e3c1e"
server_key_version: "encryptionkey@0"
I20260928 12:25:09.595690 19628 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:25:09.596444 19643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:09.596602 19628 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:25:09.596695 19628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-2/wal
uuid: "c36b7a7e5b7e48169411997e381066e9"
format_stamp: "Formatted at 2026-09-28 12:25:09 on dist-test-slave-920p"
server_key: "6fc877936d848fe3a1a2f3af47d62a98"
server_key_iv: "0cea13d3f4df28c9de2a5a297a1e3c1e"
server_key_version: "encryptionkey@0"
I20260928 12:25:09.596768 19628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:09.610177 19628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:09.611959 19628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:09.612092 19628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:09.612316 19628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:25:09.612624 19628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:25:09.612658 19628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:09.612686 19628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:25:09.612710 19628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:09.618575 19628 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.3:34299
I20260928 12:25:09.618628 19756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.3:34299 every 8 connection(s)
I20260928 12:25:09.618924 19628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
Sep 28 12:25:09 dist-test-slave-920p krb5kdc[19253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598309, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:25:09.625451 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 19628
I20260928 12:25:09.625557 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance
I20260928 12:25:09.625756 2272 external_mini_cluster.cc:1428] Setting key 45e25db947aea5c98b88d9856dfc00b2
I20260928 12:25:09.627848 19338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:09.619573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:57171 (local address 127.2.56.62:33135)
0928 12:25:09.619647 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:09.619650 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:09.619797 (+ 147us) server_negotiation.cc:407] Connection header received
0928 12:25:09.620285 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:09.620286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:09.620297 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:09.620358 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:09.621233 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.621784 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.622836 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.623071 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.623134 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:09.625437 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:09.625439 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:09.625439 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:09.625447 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:09.626804 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:09.627142 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:09.627143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:09.627144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:09.627194 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:09.627388 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:09.627389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:09.627389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:09.627482 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:09.627526 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:09.627726 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 12:25:09.627783 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 12:25:09.628034 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:09.619547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:33135 (local address 127.2.56.3:57171)
0928 12:25:09.619685 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:09.619721 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:09.620173 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:09.620417 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:09.620421 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:09.620960 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:09.621145 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:09.621149 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.621907 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.621908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:09.622739 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:09.622742 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.623157 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.623158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:09.623266 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:09.623812 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:25:09.623822 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:09.625340 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:09.626874 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:09.626879 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:09.626881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:09.627085 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:09.627244 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:09.627244 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:09.627244 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:09.627346 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:09.627522 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:09.627525 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:09.627633 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 12:25:09.627707 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 12:25:09.627825 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":77,"threads_started":1}
I20260928 12:25:09.628444 19757 heartbeater.cc:347] Connected to a master server at 127.2.56.62:33135
I20260928 12:25:09.628537 19757 heartbeater.cc:464] Registering TS with master...
I20260928 12:25:09.628703 19757 heartbeater.cc:511] Master 127.2.56.62:33135 requested a full tablet report, sending...
I20260928 12:25:09.629045 19297 ts_manager.cc:194] Registered new tserver with Master: c36b7a7e5b7e48169411997e381066e9 (127.2.56.3:34299)
I20260928 12:25:09.629438 19297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.3@KRBTEST.COM'} at 127.2.56.3:57171
I20260928 12:25:09.637408 2272 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:25:09.644294 19338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:09.638718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51142 (local address 127.2.56.62:33135)
0928 12:25:09.638774 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:09.638776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:09.638857 (+ 81us) server_negotiation.cc:407] Connection header received
0928 12:25:09.638901 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:09.638902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:09.638912 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:09.638957 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:09.639444 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.639924 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.640618 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.640795 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.640845 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:09.642146 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:09.642147 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:09.642148 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:09.642154 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:09.643323 (+ 1169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:09.643626 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:09.643626 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:09.643626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:09.643669 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:09.643798 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:09.643798 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:09.643799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:09.643933 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:09.644003 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:09.644144 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 12:25:09.644200 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 12:25:09.645910 19297 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:25:09.647437 19297 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:25:09.653576 19769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:09.649608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:38855 (local address 127.0.0.1:35290)
0928 12:25:09.649961 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:09.649969 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:09.650002 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:09.650202 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:09.650205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:09.650208 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:25:09.650284 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:09.650285 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.651282 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.651283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:09.652417 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:09.652419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.653293 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.653295 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:09.653375 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:09.653387 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 12:25:09.653448 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 12:25:09.653502 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":3,"thread_start_us":81,"threads_started":1}
I20260928 12:25:09.653586 19772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:09.649889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35290 (local address 127.2.56.2:38855)
0928 12:25:09.650035 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:09.650040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:09.650049 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:25:09.650084 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:09.650086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:09.650101 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:09.650173 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:25:09.650360 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.651172 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.652505 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.653189 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.653246 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:09.653327 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:09.653467 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 12:25:09.653504 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
I20260928 12:25:09.653820 19768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:09.649607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:34299 (local address 127.0.0.1:47938)
0928 12:25:09.649772 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:09.649810 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:09.649871 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:09.650392 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:09.650393 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:09.650395 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:25:09.650442 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:09.650443 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.651735 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.651735 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:09.652678 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:09.652680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.653607 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.653608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:09.653667 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:09.653671 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:25:09.653698 (+ 27us) client_negotiation.cc:240] Negotiation successful
0928 12:25:09.653749 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":35,"thread_start_us":64,"threads_started":1}
I20260928 12:25:09.653899 19770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:09.649610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:45651 (local address 127.0.0.1:52574)
0928 12:25:09.649772 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:09.649810 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:09.649871 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:09.650393 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:09.650393 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:09.650398 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:25:09.650440 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:09.650442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.651741 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.651742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:09.652856 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:09.652859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.653737 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.653738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:09.653788 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:09.653792 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:25:09.653816 (+ 24us) client_negotiation.cc:240] Negotiation successful
0928 12:25:09.653846 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":33,"thread_start_us":97,"threads_started":1}
I20260928 12:25:09.654136 19773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:09.650049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47938 (local address 127.2.56.3:34299)
0928 12:25:09.650262 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:09.650266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:09.650275 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:25:09.650310 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:09.650311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:09.650322 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:09.650399 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:25:09.650517 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.651616 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.652751 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.653513 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.653925 (+ 412us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:09.653998 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:09.654023 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 12:25:09.654060 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":67,"threads_started":1}
I20260928 12:25:09.654253 19771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:09.649873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52574 (local address 127.2.56.1:45651)
0928 12:25:09.650233 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:09.650237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:09.650246 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:25:09.650280 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:09.650282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:09.650297 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:09.650508 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:25:09.650585 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.651648 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.652934 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.653654 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.654035 (+ 381us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:09.654099 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:09.654121 (+ 22us) server_negotiation.cc:299] Negotiation successful
0928 12:25:09.654164 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":80,"threads_started":1}
I20260928 12:25:09.654522 19556 tablet_service.cc:1467] Processing CreateTablet for tablet 2131ca4df1584afaa1f1d7ea1c787055 (DEFAULT_TABLE table=test-table [id=d60b4aa1524a4bc4805796c0cd48c1da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:25:09.654713 19691 tablet_service.cc:1467] Processing CreateTablet for tablet 2131ca4df1584afaa1f1d7ea1c787055 (DEFAULT_TABLE table=test-table [id=d60b4aa1524a4bc4805796c0cd48c1da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:25:09.654889 19556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2131ca4df1584afaa1f1d7ea1c787055. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:09.654820 19423 tablet_service.cc:1467] Processing CreateTablet for tablet 2131ca4df1584afaa1f1d7ea1c787055 (DEFAULT_TABLE table=test-table [id=d60b4aa1524a4bc4805796c0cd48c1da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:25:09.655011 19691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2131ca4df1584afaa1f1d7ea1c787055. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:09.655092 19423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2131ca4df1584afaa1f1d7ea1c787055. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:09.657621 19774 tablet_bootstrap.cc:492] T 2131ca4df1584afaa1f1d7ea1c787055 P c36b7a7e5b7e48169411997e381066e9: Bootstrap starting.
I20260928 12:25:09.658095 19775 tablet_bootstrap.cc:492] T 2131ca4df1584afaa1f1d7ea1c787055 P 8ea665cf0fb54026920d1e33b5c985b0: Bootstrap starting.
I20260928 12:25:09.658237 19774 tablet_bootstrap.cc:654] T 2131ca4df1584afaa1f1d7ea1c787055 P c36b7a7e5b7e48169411997e381066e9: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:09.658715 19775 tablet_bootstrap.cc:654] T 2131ca4df1584afaa1f1d7ea1c787055 P 8ea665cf0fb54026920d1e33b5c985b0: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:09.658743 19774 log.cc:824] T 2131ca4df1584afaa1f1d7ea1c787055 P c36b7a7e5b7e48169411997e381066e9: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:09.659242 19775 log.cc:824] T 2131ca4df1584afaa1f1d7ea1c787055 P 8ea665cf0fb54026920d1e33b5c985b0: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:09.659464 19774 tablet_bootstrap.cc:492] T 2131ca4df1584afaa1f1d7ea1c787055 P c36b7a7e5b7e48169411997e381066e9: No bootstrap required, opened a new log
I20260928 12:25:09.659531 19774 ts_tablet_manager.cc:1397] T 2131ca4df1584afaa1f1d7ea1c787055 P c36b7a7e5b7e48169411997e381066e9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:25:09.659564 19776 tablet_bootstrap.cc:492] T 2131ca4df1584afaa1f1d7ea1c787055 P 969ac3ba7cd9488d922ae46380f7bc05: Bootstrap starting.
I20260928 12:25:09.659901 19775 tablet_bootstrap.cc:492] T 2131ca4df1584afaa1f1d7ea1c787055 P 8ea665cf0fb54026920d1e33b5c985b0: No bootstrap required, opened a new log
I20260928 12:25:09.659976 19775 ts_tablet_manager.cc:1397] T 2131ca4df1584afaa1f1d7ea1c787055 P 8ea665cf0fb54026920d1e33b5c985b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:25:09.660215 19776 tablet_bootstrap.cc:654] T 2131ca4df1584afaa1f1d7ea1c787055 P 969ac3ba7cd9488d922ae46380f7bc05: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:09.660794 19776 log.cc:824] T 2131ca4df1584afaa1f1d7ea1c787055 P 969ac3ba7cd9488d922ae46380f7bc05: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:09.661656 19774 raft_consensus.cc:348] T 2131ca4df1584afaa1f1d7ea1c787055 P c36b7a7e5b7e48169411997e381066e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c36b7a7e5b7e48169411997e381066e9" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34299 } } peers { permanent_uuid: "969ac3ba7cd9488d922ae46380f7bc05" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 38855 } } peers { permanent_uuid: "8ea665cf0fb54026920d1e33b5c985b0" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 45651 } }
I20260928 12:25:09.661696 19775 raft_consensus.cc:348] T 2131ca4df1584afaa1f1d7ea1c787055 P 8ea665cf0fb54026920d1e33b5c985b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c36b7a7e5b7e48169411997e381066e9" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34299 } } peers { permanent_uuid: "969ac3ba7cd9488d922ae46380f7bc05" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 38855 } } peers { permanent_uuid: "8ea665cf0fb54026920d1e33b5c985b0" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 45651 } }
I20260928 12:25:09.661806 19774 raft_consensus.cc:374] T 2131ca4df1584afaa1f1d7ea1c787055 P c36b7a7e5b7e48169411997e381066e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:09.661829 19774 raft_consensus.cc:729] T 2131ca4df1584afaa1f1d7ea1c787055 P c36b7a7e5b7e48169411997e381066e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c36b7a7e5b7e48169411997e381066e9, State: Initialized, Role: FOLLOWER
I20260928 12:25:09.661826 19775 raft_consensus.cc:374] T 2131ca4df1584afaa1f1d7ea1c787055 P 8ea665cf0fb54026920d1e33b5c985b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:09.661859 19775 raft_consensus.cc:729] T 2131ca4df1584afaa1f1d7ea1c787055 P 8ea665cf0fb54026920d1e33b5c985b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ea665cf0fb54026920d1e33b5c985b0, State: Initialized, Role: FOLLOWER
I20260928 12:25:09.661952 19774 consensus_queue.cc:260] T 2131ca4df1584afaa1f1d7ea1c787055 P c36b7a7e5b7e48169411997e381066e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c36b7a7e5b7e48169411997e381066e9" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34299 } } peers { permanent_uuid: "969ac3ba7cd9488d922ae46380f7bc05" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 38855 } } peers { permanent_uuid: "8ea665cf0fb54026920d1e33b5c985b0" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 45651 } }
I20260928 12:25:09.661966 19775 consensus_queue.cc:260] T 2131ca4df1584afaa1f1d7ea1c787055 P 8ea665cf0fb54026920d1e33b5c985b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c36b7a7e5b7e48169411997e381066e9" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34299 } } peers { permanent_uuid: "969ac3ba7cd9488d922ae46380f7bc05" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 38855 } } peers { permanent_uuid: "8ea665cf0fb54026920d1e33b5c985b0" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 45651 } }
I20260928 12:25:09.662218 19774 ts_tablet_manager.cc:1428] T 2131ca4df1584afaa1f1d7ea1c787055 P c36b7a7e5b7e48169411997e381066e9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:25:09.662230 19775 ts_tablet_manager.cc:1428] T 2131ca4df1584afaa1f1d7ea1c787055 P 8ea665cf0fb54026920d1e33b5c985b0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:25:09.662269 19489 heartbeater.cc:503] Master 127.2.56.62:33135 was elected leader, sending a full tablet report...
I20260928 12:25:09.662526 19757 heartbeater.cc:503] Master 127.2.56.62:33135 was elected leader, sending a full tablet report...
I20260928 12:25:09.662940 19776 tablet_bootstrap.cc:492] T 2131ca4df1584afaa1f1d7ea1c787055 P 969ac3ba7cd9488d922ae46380f7bc05: No bootstrap required, opened a new log
I20260928 12:25:09.663012 19776 ts_tablet_manager.cc:1397] T 2131ca4df1584afaa1f1d7ea1c787055 P 969ac3ba7cd9488d922ae46380f7bc05: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 12:25:09.664602 19776 raft_consensus.cc:348] T 2131ca4df1584afaa1f1d7ea1c787055 P 969ac3ba7cd9488d922ae46380f7bc05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c36b7a7e5b7e48169411997e381066e9" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34299 } } peers { permanent_uuid: "969ac3ba7cd9488d922ae46380f7bc05" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 38855 } } peers { permanent_uuid: "8ea665cf0fb54026920d1e33b5c985b0" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 45651 } }
I20260928 12:25:09.664750 19776 raft_consensus.cc:374] T 2131ca4df1584afaa1f1d7ea1c787055 P 969ac3ba7cd9488d922ae46380f7bc05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:09.664784 19776 raft_consensus.cc:729] T 2131ca4df1584afaa1f1d7ea1c787055 P 969ac3ba7cd9488d922ae46380f7bc05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 969ac3ba7cd9488d922ae46380f7bc05, State: Initialized, Role: FOLLOWER
I20260928 12:25:09.664897 19776 consensus_queue.cc:260] T 2131ca4df1584afaa1f1d7ea1c787055 P 969ac3ba7cd9488d922ae46380f7bc05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c36b7a7e5b7e48169411997e381066e9" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34299 } } peers { permanent_uuid: "969ac3ba7cd9488d922ae46380f7bc05" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 38855 } } peers { permanent_uuid: "8ea665cf0fb54026920d1e33b5c985b0" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 45651 } }
I20260928 12:25:09.665165 19776 ts_tablet_manager.cc:1428] T 2131ca4df1584afaa1f1d7ea1c787055 P 969ac3ba7cd9488d922ae46380f7bc05: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:25:09.665185 19623 heartbeater.cc:503] Master 127.2.56.62:33135 was elected leader, sending a full tablet report...
W20260928 12:25:09.720118 19624 tablet.cc:2369] T 2131ca4df1584afaa1f1d7ea1c787055 P 969ac3ba7cd9488d922ae46380f7bc05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:25:09.739231 19782 raft_consensus.cc:482] T 2131ca4df1584afaa1f1d7ea1c787055 P 969ac3ba7cd9488d922ae46380f7bc05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:25:09.739368 19782 raft_consensus.cc:504] T 2131ca4df1584afaa1f1d7ea1c787055 P 969ac3ba7cd9488d922ae46380f7bc05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c36b7a7e5b7e48169411997e381066e9" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34299 } } peers { permanent_uuid: "969ac3ba7cd9488d922ae46380f7bc05" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 38855 } } peers { permanent_uuid: "8ea665cf0fb54026920d1e33b5c985b0" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 45651 } }
I20260928 12:25:09.739719 19782 leader_election.cc:288] T 2131ca4df1584afaa1f1d7ea1c787055 P 969ac3ba7cd9488d922ae46380f7bc05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c36b7a7e5b7e48169411997e381066e9 (127.2.56.3:34299), 8ea665cf0fb54026920d1e33b5c985b0 (127.2.56.1:45651)
I20260928 12:25:09.742449 19781 raft_consensus.cc:482] T 2131ca4df1584afaa1f1d7ea1c787055 P c36b7a7e5b7e48169411997e381066e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:25:09.742566 19781 raft_consensus.cc:504] T 2131ca4df1584afaa1f1d7ea1c787055 P c36b7a7e5b7e48169411997e381066e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c36b7a7e5b7e48169411997e381066e9" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34299 } } peers { permanent_uuid: "969ac3ba7cd9488d922ae46380f7bc05" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 38855 } } peers { permanent_uuid: "8ea665cf0fb54026920d1e33b5c985b0" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 45651 } }
I20260928 12:25:09.743074 19781 leader_election.cc:288] T 2131ca4df1584afaa1f1d7ea1c787055 P c36b7a7e5b7e48169411997e381066e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 969ac3ba7cd9488d922ae46380f7bc05 (127.2.56.2:38855), 8ea665cf0fb54026920d1e33b5c985b0 (127.2.56.1:45651)
I20260928 12:25:09.743554 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:09.739890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:34299 (local address 127.2.56.2:50939)
0928 12:25:09.739935 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:09.739947 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:09.739993 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:09.740265 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:09.740265 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:09.740267 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:25:09.740364 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:09.740366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.741344 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.741345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:09.742537 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:09.742540 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.743313 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.743314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:09.743401 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:09.743403 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:25:09.743445 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 12:25:09.743486 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":28}
I20260928 12:25:09.743571 19771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:09.739958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:35579 (local address 127.2.56.1:45651)
0928 12:25:09.740016 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:09.740017 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:09.740165 (+ 148us) server_negotiation.cc:407] Connection header received
0928 12:25:09.740194 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:09.740195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:09.740206 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:09.740258 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:25:09.740462 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.741329 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.742641 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.743207 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.743249 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:09.743261 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:09.743477 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 12:25:09.743516 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 12:25:09.743775 19783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:09.739894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:45651 (local address 127.2.56.2:35579)
0928 12:25:09.740105 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:09.740114 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:09.740155 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:09.740270 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:09.740271 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:09.740272 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:25:09.740364 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:09.740366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.741383 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.741384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:09.742538 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:09.742540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.743313 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.743314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:09.743399 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:09.743403 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:25:09.743659 (+ 256us) client_negotiation.cc:240] Negotiation successful
0928 12:25:09.743697 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":84,"thread_start_us":112,"threads_started":1}
I20260928 12:25:09.743806 19773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:09.740075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:50939 (local address 127.2.56.3:34299)
0928 12:25:09.740130 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:09.740132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:09.740140 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:25:09.740171 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:09.740171 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:09.740183 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:09.740248 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:25:09.740462 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.741238 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.742632 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.743204 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.743274 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:09.743283 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:09.743713 (+ 430us) server_negotiation.cc:299] Negotiation successful
0928 12:25:09.743752 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 12:25:09.744038 19443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2131ca4df1584afaa1f1d7ea1c787055" candidate_uuid: "969ac3ba7cd9488d922ae46380f7bc05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ea665cf0fb54026920d1e33b5c985b0" is_pre_election: true
I20260928 12:25:09.744166 19443 raft_consensus.cc:2454] T 2131ca4df1584afaa1f1d7ea1c787055 P 8ea665cf0fb54026920d1e33b5c985b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 969ac3ba7cd9488d922ae46380f7bc05 in term 0.
I20260928 12:25:09.744340 19711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2131ca4df1584afaa1f1d7ea1c787055" candidate_uuid: "969ac3ba7cd9488d922ae46380f7bc05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c36b7a7e5b7e48169411997e381066e9" is_pre_election: true
I20260928 12:25:09.744418 19711 raft_consensus.cc:2454] T 2131ca4df1584afaa1f1d7ea1c787055 P c36b7a7e5b7e48169411997e381066e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 969ac3ba7cd9488d922ae46380f7bc05 in term 0.
I20260928 12:25:09.744613 19512 leader_election.cc:302] T 2131ca4df1584afaa1f1d7ea1c787055 P 969ac3ba7cd9488d922ae46380f7bc05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ea665cf0fb54026920d1e33b5c985b0, 969ac3ba7cd9488d922ae46380f7bc05; no voters:
I20260928 12:25:09.744725 19782 raft_consensus.cc:2781] T 2131ca4df1584afaa1f1d7ea1c787055 P 969ac3ba7cd9488d922ae46380f7bc05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:25:09.744769 19782 raft_consensus.cc:482] T 2131ca4df1584afaa1f1d7ea1c787055 P 969ac3ba7cd9488d922ae46380f7bc05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:25:09.744796 19782 raft_consensus.cc:3037] T 2131ca4df1584afaa1f1d7ea1c787055 P 969ac3ba7cd9488d922ae46380f7bc05 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:09.745529 19782 raft_consensus.cc:504] T 2131ca4df1584afaa1f1d7ea1c787055 P 969ac3ba7cd9488d922ae46380f7bc05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c36b7a7e5b7e48169411997e381066e9" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34299 } } peers { permanent_uuid: "969ac3ba7cd9488d922ae46380f7bc05" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 38855 } } peers { permanent_uuid: "8ea665cf0fb54026920d1e33b5c985b0" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 45651 } }
I20260928 12:25:09.745635 19782 leader_election.cc:288] T 2131ca4df1584afaa1f1d7ea1c787055 P 969ac3ba7cd9488d922ae46380f7bc05 [CANDIDATE]: Term 1 election: Requested vote from peers c36b7a7e5b7e48169411997e381066e9 (127.2.56.3:34299), 8ea665cf0fb54026920d1e33b5c985b0 (127.2.56.1:45651)
I20260928 12:25:09.745831 19443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2131ca4df1584afaa1f1d7ea1c787055" candidate_uuid: "969ac3ba7cd9488d922ae46380f7bc05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ea665cf0fb54026920d1e33b5c985b0"
I20260928 12:25:09.745831 19711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2131ca4df1584afaa1f1d7ea1c787055" candidate_uuid: "969ac3ba7cd9488d922ae46380f7bc05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c36b7a7e5b7e48169411997e381066e9"
I20260928 12:25:09.745900 19711 raft_consensus.cc:3037] T 2131ca4df1584afaa1f1d7ea1c787055 P c36b7a7e5b7e48169411997e381066e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:09.745901 19443 raft_consensus.cc:3037] T 2131ca4df1584afaa1f1d7ea1c787055 P 8ea665cf0fb54026920d1e33b5c985b0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:09.746665 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:09.742996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:45651 (local address 127.2.56.3:45363)
0928 12:25:09.743066 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:09.743075 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:09.743111 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:09.743561 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:09.743562 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:09.743563 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:25:09.743620 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:09.743622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.744489 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.744490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:09.745646 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:09.745648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.746423 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.746424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:09.746494 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:09.746497 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:25:09.746535 (+ 38us) client_negotiation.cc:240] Negotiation successful
0928 12:25:09.746598 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 12:25:09.746708 19711 raft_consensus.cc:2454] T 2131ca4df1584afaa1f1d7ea1c787055 P c36b7a7e5b7e48169411997e381066e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 969ac3ba7cd9488d922ae46380f7bc05 in term 1.
I20260928 12:25:09.746712 19443 raft_consensus.cc:2454] T 2131ca4df1584afaa1f1d7ea1c787055 P 8ea665cf0fb54026920d1e33b5c985b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 969ac3ba7cd9488d922ae46380f7bc05 in term 1.
I20260928 12:25:09.746872 19772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:09.743309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:50213 (local address 127.2.56.2:38855)
0928 12:25:09.743352 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:09.743354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:09.743434 (+ 80us) server_negotiation.cc:407] Connection header received
0928 12:25:09.743522 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:09.743523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:09.743532 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:09.743558 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:25:09.744163 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.744791 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.746059 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.746557 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.746590 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:09.746599 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:09.746787 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 12:25:09.746820 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:25:09.746943 19785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:09.743187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:45363 (local address 127.2.56.1:45651)
0928 12:25:09.743415 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:09.743417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:09.743423 (+ 6us) server_negotiation.cc:407] Connection header received
0928 12:25:09.743444 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:09.743445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:09.743454 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:09.743683 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:25:09.743714 (+ 31us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.744402 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.745738 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.746342 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.746812 (+ 470us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:09.746825 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:09.746841 (+ 16us) server_negotiation.cc:299] Negotiation successful
0928 12:25:09.746875 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":79,"threads_started":1}
I20260928 12:25:09.746934 19513 leader_election.cc:302] T 2131ca4df1584afaa1f1d7ea1c787055 P 969ac3ba7cd9488d922ae46380f7bc05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 969ac3ba7cd9488d922ae46380f7bc05, c36b7a7e5b7e48169411997e381066e9; no voters:
I20260928 12:25:09.747043 19782 raft_consensus.cc:2781] T 2131ca4df1584afaa1f1d7ea1c787055 P 969ac3ba7cd9488d922ae46380f7bc05 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:25:09.747089 19784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:09.742996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:38855 (local address 127.2.56.3:50213)
0928 12:25:09.743375 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:09.743384 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:09.743456 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:09.744028 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:09.744029 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:09.744031 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:25:09.744081 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:09.744082 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.744878 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.744878 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:09.745951 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:09.745953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.746648 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.746649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:09.746711 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:09.746714 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:25:09.746969 (+ 255us) client_negotiation.cc:240] Negotiation successful
0928 12:25:09.747007 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":159,"thread_start_us":106,"threads_started":1}
I20260928 12:25:09.747167 19782 raft_consensus.cc:686] T 2131ca4df1584afaa1f1d7ea1c787055 P 969ac3ba7cd9488d922ae46380f7bc05 [term 1 LEADER]: Becoming Leader. State: Replica: 969ac3ba7cd9488d922ae46380f7bc05, State: Running, Role: LEADER
I20260928 12:25:09.747198 19443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2131ca4df1584afaa1f1d7ea1c787055" candidate_uuid: "c36b7a7e5b7e48169411997e381066e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ea665cf0fb54026920d1e33b5c985b0" is_pre_election: true
I20260928 12:25:09.747273 19443 raft_consensus.cc:2379] T 2131ca4df1584afaa1f1d7ea1c787055 P 8ea665cf0fb54026920d1e33b5c985b0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c36b7a7e5b7e48169411997e381066e9 in current term 1: Already voted for candidate 969ac3ba7cd9488d922ae46380f7bc05 in this term.
I20260928 12:25:09.747279 19782 consensus_queue.cc:237] T 2131ca4df1584afaa1f1d7ea1c787055 P 969ac3ba7cd9488d922ae46380f7bc05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c36b7a7e5b7e48169411997e381066e9" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34299 } } peers { permanent_uuid: "969ac3ba7cd9488d922ae46380f7bc05" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 38855 } } peers { permanent_uuid: "8ea665cf0fb54026920d1e33b5c985b0" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 45651 } }
I20260928 12:25:09.747331 19577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2131ca4df1584afaa1f1d7ea1c787055" candidate_uuid: "c36b7a7e5b7e48169411997e381066e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "969ac3ba7cd9488d922ae46380f7bc05" is_pre_election: true
I20260928 12:25:09.747625 19645 leader_election.cc:302] T 2131ca4df1584afaa1f1d7ea1c787055 P c36b7a7e5b7e48169411997e381066e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c36b7a7e5b7e48169411997e381066e9; no voters: 8ea665cf0fb54026920d1e33b5c985b0, 969ac3ba7cd9488d922ae46380f7bc05
I20260928 12:25:09.747795 19781 raft_consensus.cc:2726] T 2131ca4df1584afaa1f1d7ea1c787055 P c36b7a7e5b7e48169411997e381066e9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 12:25:09.748066 19298 catalog_manager.cc:5495] T 2131ca4df1584afaa1f1d7ea1c787055 P 969ac3ba7cd9488d922ae46380f7bc05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 969ac3ba7cd9488d922ae46380f7bc05 (127.2.56.2). New cstate: current_term: 1 leader_uuid: "969ac3ba7cd9488d922ae46380f7bc05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c36b7a7e5b7e48169411997e381066e9" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "969ac3ba7cd9488d922ae46380f7bc05" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 38855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ea665cf0fb54026920d1e33b5c985b0" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 45651 } health_report { overall_health: UNKNOWN } } }
I20260928 12:25:09.769053 19772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:09.766378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35304 (local address 127.2.56.2:38855)
0928 12:25:09.766431 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:09.766433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:09.766442 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:25:09.766499 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:09.766500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:09.766518 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:09.766574 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:25:09.766707 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.767445 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.768297 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.768484 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.768537 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:09.768695 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:25:09.768798 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:25:09.768870 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:09.768927 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 12:25:09.768949 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 12:25:09.770840 19443 raft_consensus.cc:1261] T 2131ca4df1584afaa1f1d7ea1c787055 P 8ea665cf0fb54026920d1e33b5c985b0 [term 1 FOLLOWER]: Refusing update from remote peer 969ac3ba7cd9488d922ae46380f7bc05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:25:09.770861 19711 raft_consensus.cc:1261] T 2131ca4df1584afaa1f1d7ea1c787055 P c36b7a7e5b7e48169411997e381066e9 [term 1 FOLLOWER]: Refusing update from remote peer 969ac3ba7cd9488d922ae46380f7bc05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:25:09.771143 19786 consensus_queue.cc:1035] T 2131ca4df1584afaa1f1d7ea1c787055 P 969ac3ba7cd9488d922ae46380f7bc05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c36b7a7e5b7e48169411997e381066e9" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 34299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:25:09.771268 19782 consensus_queue.cc:1035] T 2131ca4df1584afaa1f1d7ea1c787055 P 969ac3ba7cd9488d922ae46380f7bc05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ea665cf0fb54026920d1e33b5c985b0" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 45651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:25:09.773434 19789 mvcc.cc:205] Tried to move back new op lower bound from 7334290676818698240 to 7334290676725325824. Current Snapshot: MvccSnapshot[applied={T|T < 7334290676818698240}]
I20260928 12:25:09.774135 19790 mvcc.cc:205] Tried to move back new op lower bound from 7334290676818698240 to 7334290676725325824. Current Snapshot: MvccSnapshot[applied={T|T < 7334290676818698240}]
I20260928 12:25:09.774248 19791 mvcc.cc:205] Tried to move back new op lower bound from 7334290676818698240 to 7334290676725325824. Current Snapshot: MvccSnapshot[applied={T|T < 7334290676818698240}]
I20260928 12:25:09.775985 19297 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:25:09.776087 19297 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:25:09.776770 19297 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cae95ebd8ef344d28b0f545d0ce6895b: Sending DeleteTablet for 3 replicas of tablet 2131ca4df1584afaa1f1d7ea1c787055
I20260928 12:25:09.777108 19691 tablet_service.cc:1514] Processing DeleteTablet for tablet 2131ca4df1584afaa1f1d7ea1c787055 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:25:09 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:47938
I20260928 12:25:09.777129 19556 tablet_service.cc:1514] Processing DeleteTablet for tablet 2131ca4df1584afaa1f1d7ea1c787055 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:25:09 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:35290
I20260928 12:25:09.777148 19423 tablet_service.cc:1514] Processing DeleteTablet for tablet 2131ca4df1584afaa1f1d7ea1c787055 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:25:09 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:52574
I20260928 12:25:09.777355 19801 tablet_replica.cc:330] T 2131ca4df1584afaa1f1d7ea1c787055 P 8ea665cf0fb54026920d1e33b5c985b0: stopping tablet replica
I20260928 12:25:09.777452 19801 raft_consensus.cc:2229] T 2131ca4df1584afaa1f1d7ea1c787055 P 8ea665cf0fb54026920d1e33b5c985b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:25:09.777534 19800 tablet_replica.cc:330] T 2131ca4df1584afaa1f1d7ea1c787055 P 969ac3ba7cd9488d922ae46380f7bc05: stopping tablet replica
I20260928 12:25:09.777575 19801 raft_consensus.cc:2258] T 2131ca4df1584afaa1f1d7ea1c787055 P 8ea665cf0fb54026920d1e33b5c985b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:25:09.777647 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 19360
I20260928 12:25:09.778301 19799 tablet_replica.cc:330] T 2131ca4df1584afaa1f1d7ea1c787055 P c36b7a7e5b7e48169411997e381066e9: stopping tablet replica
I20260928 12:25:09.778395 19799 raft_consensus.cc:2229] T 2131ca4df1584afaa1f1d7ea1c787055 P c36b7a7e5b7e48169411997e381066e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:25:09.778430 19800 raft_consensus.cc:2229] T 2131ca4df1584afaa1f1d7ea1c787055 P 969ac3ba7cd9488d922ae46380f7bc05 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:25:09.778491 19799 raft_consensus.cc:2258] T 2131ca4df1584afaa1f1d7ea1c787055 P c36b7a7e5b7e48169411997e381066e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:25:09.778566 19800 raft_consensus.cc:2258] T 2131ca4df1584afaa1f1d7ea1c787055 P 969ac3ba7cd9488d922ae46380f7bc05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:25:09.778959 19799 ts_tablet_manager.cc:1905] T 2131ca4df1584afaa1f1d7ea1c787055 P c36b7a7e5b7e48169411997e381066e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:25:09.779029 19800 ts_tablet_manager.cc:1905] T 2131ca4df1584afaa1f1d7ea1c787055 P 969ac3ba7cd9488d922ae46380f7bc05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:25:09.780779 19799 ts_tablet_manager.cc:1918] T 2131ca4df1584afaa1f1d7ea1c787055 P c36b7a7e5b7e48169411997e381066e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:25:09.780779 19800 ts_tablet_manager.cc:1918] T 2131ca4df1584afaa1f1d7ea1c787055 P 969ac3ba7cd9488d922ae46380f7bc05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:25:09.780857 19800 log.cc:1196] T 2131ca4df1584afaa1f1d7ea1c787055 P 969ac3ba7cd9488d922ae46380f7bc05: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-1/wal/wals/2131ca4df1584afaa1f1d7ea1c787055
I20260928 12:25:09.780857 19799 log.cc:1196] T 2131ca4df1584afaa1f1d7ea1c787055 P c36b7a7e5b7e48169411997e381066e9: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598072638673-2272-0/minicluster-data/ts-2/wal/wals/2131ca4df1584afaa1f1d7ea1c787055
I20260928 12:25:09.781123 19799 ts_tablet_manager.cc:1939] T 2131ca4df1584afaa1f1d7ea1c787055 P c36b7a7e5b7e48169411997e381066e9: Deleting consensus metadata
I20260928 12:25:09.781147 19800 ts_tablet_manager.cc:1939] T 2131ca4df1584afaa1f1d7ea1c787055 P 969ac3ba7cd9488d922ae46380f7bc05: Deleting consensus metadata
I20260928 12:25:09.781744 19283 catalog_manager.cc:4841] TS 969ac3ba7cd9488d922ae46380f7bc05 (127.2.56.2:38855): tablet 2131ca4df1584afaa1f1d7ea1c787055 (table test-table [id=d60b4aa1524a4bc4805796c0cd48c1da]) successfully deleted
I20260928 12:25:09.781744 19285 catalog_manager.cc:4841] TS c36b7a7e5b7e48169411997e381066e9 (127.2.56.3:34299): tablet 2131ca4df1584afaa1f1d7ea1c787055 (table test-table [id=d60b4aa1524a4bc4805796c0cd48c1da]) successfully deleted
W20260928 12:25:09.784823 19284 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.56.1:45651 (error 108)
I20260928 12:25:09.785089 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 19494
I20260928 12:25:09.785681 19770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:09.785205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:45651 (local address 127.0.0.1:52586)
0928 12:25:09.785608 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:09.785647 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.1:45651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382}
W20260928 12:25:09.788503 19284 catalog_manager.cc:4584] TS 8ea665cf0fb54026920d1e33b5c985b0 (127.2.56.1:45651): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2131ca4df1584afaa1f1d7ea1c787055: Network error: Client connection negotiation failed: client connection to 127.2.56.1:45651: connect: Connection refused (error 111)
I20260928 12:25:09.791059 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 19628
I20260928 12:25:09.794986 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 19266
2026-09-28T12:25:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1947 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T12:25:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:25:09Z Disabled control of system clock
I20260928 12:25:09.820272 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:44791
--webserver_interface=127.2.56.62
--webserver_port=0
--builtin_ntp_servers=127.2.56.20:45011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:44791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 12:25:09.900472 19806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:09.900681 19806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:09.900700 19806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:09.903954 19806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:25:09.904023 19806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:09.904035 19806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:09.904049 19806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:25:09.904062 19806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:25:09.906064 19806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:45011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:44791
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:44791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:09.906296 19806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:09.906567 19806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:09.911173 19812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:09.911201 19811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:09.911324 19806 server_base.cc:1034] running on GCE node
W20260928 12:25:09.911181 19814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:09.911643 19806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:09.911932 19806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:09.913089 19806 hybrid_clock.cc:648] HybridClock initialized: now 1790598309913073 us; error 27 us; skew 500 ppm
I20260928 12:25:09.914292 19806 webserver.cc:460] Webserver started at http://127.2.56.62:36865/ using document root <none> and password file <none>
I20260928 12:25:09.914532 19806 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:09.914583 19806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:09.914710 19806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:09.915733 19806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "46421a3cfca94cc4a42c5a33caddb729"
format_stamp: "Formatted at 2026-09-28 12:25:09 on dist-test-slave-920p"
server_key: "f12bdee10b2534a2125483a7dab0c7a4"
server_key_iv: "68734075f53192d51489d8dafb9ef74d"
server_key_version: "encryptionkey@0"
I20260928 12:25:09.916052 19806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "46421a3cfca94cc4a42c5a33caddb729"
format_stamp: "Formatted at 2026-09-28 12:25:09 on dist-test-slave-920p"
server_key: "f12bdee10b2534a2125483a7dab0c7a4"
server_key_iv: "68734075f53192d51489d8dafb9ef74d"
server_key_version: "encryptionkey@0"
I20260928 12:25:09.917209 19806 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:25:09.917934 19820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:09.918071 19806 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:25:09.918154 19806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "46421a3cfca94cc4a42c5a33caddb729"
format_stamp: "Formatted at 2026-09-28 12:25:09 on dist-test-slave-920p"
server_key: "f12bdee10b2534a2125483a7dab0c7a4"
server_key_iv: "68734075f53192d51489d8dafb9ef74d"
server_key_version: "encryptionkey@0"
I20260928 12:25:09.918223 19806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:09.927292 19806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:09.927546 19806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:09.927661 19806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:09.931288 19806 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:44791
I20260928 12:25:09.931316 19872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:44791 every 8 connection(s)
I20260928 12:25:09.931617 19806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:25:09.932489 19873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:09.934301 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 19806
I20260928 12:25:09.934397 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:25:09.934566 2272 external_mini_cluster.cc:1428] Setting key db01f4cb210f1e88387ea98df09aed8e
I20260928 12:25:09.934952 19873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46421a3cfca94cc4a42c5a33caddb729: Bootstrap starting.
I20260928 12:25:09.935611 19873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46421a3cfca94cc4a42c5a33caddb729: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:09.935927 19873 log.cc:824] T 00000000000000000000000000000000 P 46421a3cfca94cc4a42c5a33caddb729: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:09.936693 19873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46421a3cfca94cc4a42c5a33caddb729: No bootstrap required, opened a new log
I20260928 12:25:09.938854 19873 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46421a3cfca94cc4a42c5a33caddb729 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46421a3cfca94cc4a42c5a33caddb729" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 44791 } }
I20260928 12:25:09.939025 19873 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46421a3cfca94cc4a42c5a33caddb729 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:09.939079 19873 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46421a3cfca94cc4a42c5a33caddb729 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46421a3cfca94cc4a42c5a33caddb729, State: Initialized, Role: FOLLOWER
I20260928 12:25:09.939239 19873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46421a3cfca94cc4a42c5a33caddb729 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46421a3cfca94cc4a42c5a33caddb729" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 44791 } }
I20260928 12:25:09.939320 19873 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46421a3cfca94cc4a42c5a33caddb729 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:25:09.939350 19873 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46421a3cfca94cc4a42c5a33caddb729 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:25:09.939392 19873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46421a3cfca94cc4a42c5a33caddb729 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:09.939998 19873 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46421a3cfca94cc4a42c5a33caddb729 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46421a3cfca94cc4a42c5a33caddb729" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 44791 } }
I20260928 12:25:09.940104 19873 leader_election.cc:302] T 00000000000000000000000000000000 P 46421a3cfca94cc4a42c5a33caddb729 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46421a3cfca94cc4a42c5a33caddb729; no voters:
I20260928 12:25:09.940270 19873 leader_election.cc:288] T 00000000000000000000000000000000 P 46421a3cfca94cc4a42c5a33caddb729 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:25:09.940328 19878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46421a3cfca94cc4a42c5a33caddb729 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:25:09.940519 19878 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46421a3cfca94cc4a42c5a33caddb729 [term 1 LEADER]: Becoming Leader. State: Replica: 46421a3cfca94cc4a42c5a33caddb729, State: Running, Role: LEADER
I20260928 12:25:09.940591 19876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:09.935078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53420 (local address 127.2.56.62:44791)
0928 12:25:09.935281 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:09.935291 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:09.935324 (+ 33us) server_negotiation.cc:407] Connection header received
0928 12:25:09.935873 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:09.935882 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:09.935960 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:09.936117 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:09.936319 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.937676 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.938461 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:09.939069 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:09.939192 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:09.939832 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:09.939836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:09.939836 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:09.940058 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:09.940104 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:09.940119 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:09.940185 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:09.940299 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 12:25:09.940381 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
I20260928 12:25:09.940675 19873 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46421a3cfca94cc4a42c5a33caddb729 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:25:09.940632 19878 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46421a3cfca94cc4a42c5a33caddb729 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46421a3cfca94cc4a42c5a33caddb729" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 44791 } }
I20260928 12:25:09.941375 19879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46421a3cfca94cc4a42c5a33caddb729 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46421a3cfca94cc4a42c5a33caddb729" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46421a3cfca94cc4a42c5a33caddb729" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 44791 } } }
I20260928 12:25:09.941457 19879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46421a3cfca94cc4a42c5a33caddb729 [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:09.941680 19888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:25:09.941813 19880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46421a3cfca94cc4a42c5a33caddb729 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46421a3cfca94cc4a42c5a33caddb729. Latest consensus state: current_term: 1 leader_uuid: "46421a3cfca94cc4a42c5a33caddb729" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46421a3cfca94cc4a42c5a33caddb729" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 44791 } } }
I20260928 12:25:09.941885 19880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46421a3cfca94cc4a42c5a33caddb729 [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:09.942283 19888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:25:09.943699 19888 catalog_manager.cc:1347] Generated new cluster ID: 69eda759dcaa4580b495fca96a2d055e
I20260928 12:25:09.943754 19888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:25:09.962882 19888 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:25:09.963387 19888 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:25:09.969327 19888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46421a3cfca94cc4a42c5a33caddb729: Generated new TSK 0
I20260928 12:25:09.969496 19888 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:25:09.979342 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.1:0
--local_ip_for_outbound_sockets=127.2.56.1
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:44791
--builtin_ntp_servers=127.2.56.20:45011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 12:25:10.059912 19897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:10.060102 19897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:10.060122 19897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:10.061249 19897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:10.061304 19897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:10.061329 19897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.1
I20260928 12:25:10.062744 19897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:45011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:44791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.1
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:10.062961 19897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:10.063234 19897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:10.067821 19903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:10.067816 19902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:10.067919 19897 server_base.cc:1034] running on GCE node
W20260928 12:25:10.068228 19905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:10.068416 19897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:10.068665 19897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:10.069802 19897 hybrid_clock.cc:648] HybridClock initialized: now 1790598310069792 us; error 35 us; skew 500 ppm
I20260928 12:25:10.070974 19897 webserver.cc:460] Webserver started at http://127.2.56.1:37941/ using document root <none> and password file <none>
I20260928 12:25:10.071197 19897 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:10.071239 19897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:10.071357 19897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:10.072388 19897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-0/data/instance:
uuid: "9e83769fb7d846bf855cea40798ebcdf"
format_stamp: "Formatted at 2026-09-28 12:25:10 on dist-test-slave-920p"
server_key: "be83057ff32e0fc69109db8a6e3756d5"
server_key_iv: "e7ea3c3379280eb2444eab2096be86a1"
server_key_version: "encryptionkey@0"
I20260928 12:25:10.072708 19897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance:
uuid: "9e83769fb7d846bf855cea40798ebcdf"
format_stamp: "Formatted at 2026-09-28 12:25:10 on dist-test-slave-920p"
server_key: "be83057ff32e0fc69109db8a6e3756d5"
server_key_iv: "e7ea3c3379280eb2444eab2096be86a1"
server_key_version: "encryptionkey@0"
I20260928 12:25:10.073953 19897 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:25:10.074671 19911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:10.074838 19897 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:25:10.074918 19897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-0/wal
uuid: "9e83769fb7d846bf855cea40798ebcdf"
format_stamp: "Formatted at 2026-09-28 12:25:10 on dist-test-slave-920p"
server_key: "be83057ff32e0fc69109db8a6e3756d5"
server_key_iv: "e7ea3c3379280eb2444eab2096be86a1"
server_key_version: "encryptionkey@0"
I20260928 12:25:10.074985 19897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:10.083076 19897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:10.083325 19897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:10.083442 19897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:10.083634 19897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:25:10.083918 19897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:25:10.083950 19897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:10.083976 19897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:25:10.084000 19897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:10.089332 19897 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.1:37253
I20260928 12:25:10.089370 20024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.1:37253 every 8 connection(s)
I20260928 12:25:10.089670 19897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
I20260928 12:25:10.094134 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 19897
I20260928 12:25:10.094214 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance
I20260928 12:25:10.094383 2272 external_mini_cluster.cc:1428] Setting key 94a92f55d90425ecbb23f1a0441d7cff
I20260928 12:25:10.094444 19876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:10.090317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:39013 (local address 127.2.56.62:44791)
0928 12:25:10.090370 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:10.090372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:10.090582 (+ 210us) server_negotiation.cc:407] Connection header received
0928 12:25:10.091066 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:10.091067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:10.091079 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:10.091122 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:10.091737 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:10.092239 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:10.093147 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:10.093394 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:10.093445 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:10.094172 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:10.094174 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:10.094174 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:10.094179 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:10.094194 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:10.094198 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:10.094242 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:10.094335 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 12:25:10.094366 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 12:25:10.094624 20027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:10.090304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:44791 (local address 127.2.56.1:39013)
0928 12:25:10.090472 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:10.090503 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:10.090975 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:10.091185 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:10.091188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:10.091196 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:10.091659 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:10.091663 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:10.092321 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:10.092322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:10.093071 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:10.093073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:10.093468 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:10.093469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:10.093586 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:10.094019 (+ 433us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:25:10.094019 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:10.094068 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:25:10.094089 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:10.094262 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:10.094267 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:10.094270 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:25:10.094327 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 12:25:10.094424 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20260928 12:25:10.095047 20025 heartbeater.cc:347] Connected to a master server at 127.2.56.62:44791
I20260928 12:25:10.095136 20025 heartbeater.cc:464] Registering TS with master...
I20260928 12:25:10.095296 20025 heartbeater.cc:511] Master 127.2.56.62:44791 requested a full tablet report, sending...
I20260928 12:25:10.095302 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.2:0
--local_ip_for_outbound_sockets=127.2.56.2
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:44791
--builtin_ntp_servers=127.2.56.20:45011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 12:25:10.095774 19836 ts_manager.cc:194] Registered new tserver with Master: 9e83769fb7d846bf855cea40798ebcdf (127.2.56.1:37253)
I20260928 12:25:10.096359 19836 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.56.1:39013
W20260928 12:25:10.173256 20028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:10.173501 20028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:10.173524 20028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:10.174652 20028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:10.174688 20028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:10.174710 20028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.2
I20260928 12:25:10.175998 20028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:45011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:44791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.2
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:10.176196 20028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:10.176443 20028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:10.181015 20036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:10.181073 20034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:10.181027 20033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:10.181305 20028 server_base.cc:1034] running on GCE node
I20260928 12:25:10.181488 20028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:10.181665 20028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:10.182806 20028 hybrid_clock.cc:648] HybridClock initialized: now 1790598310182790 us; error 26 us; skew 500 ppm
I20260928 12:25:10.183954 20028 webserver.cc:460] Webserver started at http://127.2.56.2:43839/ using document root <none> and password file <none>
I20260928 12:25:10.184157 20028 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:10.184208 20028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:10.184333 20028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:10.185396 20028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-1/data/instance:
uuid: "0095203c0d004c17b2bad8f110984edb"
format_stamp: "Formatted at 2026-09-28 12:25:10 on dist-test-slave-920p"
server_key: "eae4dedc009830be81b37879f0e172c7"
server_key_iv: "f791615e9827a5502b97900045b7f995"
server_key_version: "encryptionkey@0"
I20260928 12:25:10.185698 20028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance:
uuid: "0095203c0d004c17b2bad8f110984edb"
format_stamp: "Formatted at 2026-09-28 12:25:10 on dist-test-slave-920p"
server_key: "eae4dedc009830be81b37879f0e172c7"
server_key_iv: "f791615e9827a5502b97900045b7f995"
server_key_version: "encryptionkey@0"
I20260928 12:25:10.186807 20028 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:25:10.187420 20042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:10.187573 20028 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:25:10.187650 20028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-1/wal
uuid: "0095203c0d004c17b2bad8f110984edb"
format_stamp: "Formatted at 2026-09-28 12:25:10 on dist-test-slave-920p"
server_key: "eae4dedc009830be81b37879f0e172c7"
server_key_iv: "f791615e9827a5502b97900045b7f995"
server_key_version: "encryptionkey@0"
I20260928 12:25:10.187704 20028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:10.197079 20028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:10.197350 20028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:10.197465 20028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:10.197639 20028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:25:10.197926 20028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:25:10.197958 20028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:10.197983 20028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:25:10.198007 20028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:10.203150 20028 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.2:35017
I20260928 12:25:10.203212 20155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.2:35017 every 8 connection(s)
I20260928 12:25:10.203469 20028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
I20260928 12:25:10.208205 19876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:10.204064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:38775 (local address 127.2.56.62:44791)
0928 12:25:10.204136 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:10.204138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:10.204332 (+ 194us) server_negotiation.cc:407] Connection header received
0928 12:25:10.204776 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:10.204777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:10.204787 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:10.204836 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:10.205471 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:10.205968 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:10.206890 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:10.207094 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:10.207158 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:10.207919 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:10.207921 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:10.207921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:10.207925 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:10.207939 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:10.207943 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:10.207999 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:10.208116 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 12:25:10.208143 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 12:25:10.208400 20158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:10.204059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:44791 (local address 127.2.56.2:38775)
0928 12:25:10.204215 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:10.204247 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:10.204700 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:10.204885 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:10.204888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:10.204895 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:10.205403 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:10.205407 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:10.206043 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:10.206044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:10.206803 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:10.206805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:10.207166 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:10.207167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:10.207267 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:10.207765 (+ 498us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:25:10.207765 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:10.207792 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:25:10.207814 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:10.208025 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:10.208031 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:10.208034 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:25:10.208110 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 12:25:10.208194 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":80,"threads_started":1}
I20260928 12:25:10.208846 20156 heartbeater.cc:347] Connected to a master server at 127.2.56.62:44791
I20260928 12:25:10.208928 20156 heartbeater.cc:464] Registering TS with master...
I20260928 12:25:10.209074 20156 heartbeater.cc:511] Master 127.2.56.62:44791 requested a full tablet report, sending...
I20260928 12:25:10.209411 19836 ts_manager.cc:194] Registered new tserver with Master: 0095203c0d004c17b2bad8f110984edb (127.2.56.2:35017)
I20260928 12:25:10.209476 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 20028
I20260928 12:25:10.209568 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance
I20260928 12:25:10.209755 2272 external_mini_cluster.cc:1428] Setting key c0cef4f62ab21a94ab995253dacb58ed
I20260928 12:25:10.209826 19836 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.56.2:38775
I20260928 12:25:10.210747 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.3:0
--local_ip_for_outbound_sockets=127.2.56.3
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:44791
--builtin_ntp_servers=127.2.56.20:45011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 12:25:10.287250 20159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:10.287428 20159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:10.287448 20159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:10.288498 20159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:10.288530 20159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:10.288555 20159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.3
I20260928 12:25:10.289853 20159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:45011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:44791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.3
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:10.290040 20159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:10.290275 20159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:10.294752 20167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:10.294782 20164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:10.294801 20165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:10.294942 20159 server_base.cc:1034] running on GCE node
I20260928 12:25:10.295116 20159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:10.295307 20159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:10.296464 20159 hybrid_clock.cc:648] HybridClock initialized: now 1790598310296443 us; error 34 us; skew 500 ppm
I20260928 12:25:10.297760 20159 webserver.cc:460] Webserver started at http://127.2.56.3:43305/ using document root <none> and password file <none>
I20260928 12:25:10.297982 20159 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:10.298026 20159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:10.298149 20159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:10.299149 20159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-2/data/instance:
uuid: "3baf8a3cf0ce42469c4d2d62e7c8a75d"
format_stamp: "Formatted at 2026-09-28 12:25:10 on dist-test-slave-920p"
server_key: "9c86def483565c355da0734c7eb45278"
server_key_iv: "6d42804d0289fa5470e980dfd6897af0"
server_key_version: "encryptionkey@0"
I20260928 12:25:10.299769 20159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance:
uuid: "3baf8a3cf0ce42469c4d2d62e7c8a75d"
format_stamp: "Formatted at 2026-09-28 12:25:10 on dist-test-slave-920p"
server_key: "9c86def483565c355da0734c7eb45278"
server_key_iv: "6d42804d0289fa5470e980dfd6897af0"
server_key_version: "encryptionkey@0"
I20260928 12:25:10.302241 20159 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260928 12:25:10.303192 20173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:10.303421 20159 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:25:10.303509 20159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-2/wal
uuid: "3baf8a3cf0ce42469c4d2d62e7c8a75d"
format_stamp: "Formatted at 2026-09-28 12:25:10 on dist-test-slave-920p"
server_key: "9c86def483565c355da0734c7eb45278"
server_key_iv: "6d42804d0289fa5470e980dfd6897af0"
server_key_version: "encryptionkey@0"
I20260928 12:25:10.303710 20159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:10.316840 20159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:10.317137 20159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:10.317252 20159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:10.317478 20159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:25:10.317772 20159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:25:10.317806 20159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:10.317832 20159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:25:10.317857 20159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:10.323133 20159 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.3:43477
I20260928 12:25:10.323192 20286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.3:43477 every 8 connection(s)
I20260928 12:25:10.323460 20159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
I20260928 12:25:10.324821 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 20159
I20260928 12:25:10.324908 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance
I20260928 12:25:10.325088 2272 external_mini_cluster.cc:1428] Setting key b6acf4dea97c761f778a5966549e7852
I20260928 12:25:10.328135 19876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:10.324095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:56867 (local address 127.2.56.62:44791)
0928 12:25:10.324151 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:10.324153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:10.324338 (+ 185us) server_negotiation.cc:407] Connection header received
0928 12:25:10.324830 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:10.324832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:10.324843 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:10.324891 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:10.325646 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:10.326239 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:10.327125 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:10.327300 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:10.327340 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:10.327958 (+ 618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:10.327959 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:10.327960 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:10.327963 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:10.327976 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:10.327980 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:10.328015 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:10.328060 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 12:25:10.328080 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 12:25:10.328346 20289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:10.324070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:44791 (local address 127.2.56.3:56867)
0928 12:25:10.324231 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:10.324269 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:10.324731 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:10.324960 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:10.324964 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:10.324973 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:10.325572 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:10.325578 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:10.326310 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:10.326311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:10.327059 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:10.327061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:10.327349 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:10.327350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:10.327455 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:10.327851 (+ 396us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:25:10.327851 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:10.327877 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:25:10.327896 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:10.328015 (+ 119us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:10.328017 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:10.328019 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:25:10.328059 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 12:25:10.328147 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":72,"threads_started":1}
I20260928 12:25:10.328753 20287 heartbeater.cc:347] Connected to a master server at 127.2.56.62:44791
I20260928 12:25:10.328823 20287 heartbeater.cc:464] Registering TS with master...
I20260928 12:25:10.328982 20287 heartbeater.cc:511] Master 127.2.56.62:44791 requested a full tablet report, sending...
I20260928 12:25:10.329308 19836 ts_manager.cc:194] Registered new tserver with Master: 3baf8a3cf0ce42469c4d2d62e7c8a75d (127.2.56.3:43477)
I20260928 12:25:10.329653 19836 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.56.3:56867
I20260928 12:25:10.336567 2272 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:25:10.340335 19876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:10.337836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53432 (local address 127.2.56.62:44791)
0928 12:25:10.337891 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:10.337893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:10.337983 (+ 90us) server_negotiation.cc:407] Connection header received
0928 12:25:10.338060 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:10.338061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:10.338068 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:10.338118 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:10.338234 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:10.338677 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:10.339384 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:10.339549 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:10.339598 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:10.340164 (+ 566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:10.340165 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:10.340165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:10.340169 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:10.340179 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:10.340182 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:10.340214 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:10.340258 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 12:25:10.340274 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 12:25:10.341728 19836 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53432:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:25:10.343256 19836 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:25:10.349005 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:10.345024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:35017 (local address 127.0.0.1:56594)
0928 12:25:10.345606 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:10.345614 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:10.345683 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:10.345916 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:10.345917 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:10.345919 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:25:10.345980 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:10.345982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:10.347143 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:10.347143 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:10.347940 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:10.347942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:10.348740 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:10.348741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:10.348812 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:10.348825 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 12:25:10.348887 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 12:25:10.348934 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":27,"thread_start_us":84,"threads_started":1}
I20260928 12:25:10.349015 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:10.345214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56594 (local address 127.2.56.2:35017)
0928 12:25:10.345415 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:10.345419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:10.345659 (+ 240us) server_negotiation.cc:407] Connection header received
0928 12:25:10.345763 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:10.345765 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:10.345781 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:10.345851 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:25:10.346074 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:10.347073 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:10.347996 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:10.348658 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:10.348729 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:10.348816 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:10.348898 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 12:25:10.348936 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":99,"threads_started":1}
I20260928 12:25:10.349134 20299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:10.345388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56016 (local address 127.2.56.1:37253)
0928 12:25:10.345518 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:10.345523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:10.345646 (+ 123us) server_negotiation.cc:407] Connection header received
0928 12:25:10.345762 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:10.345764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:10.345778 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:10.345846 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:25:10.346092 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:10.346760 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:10.348229 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:10.348704 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:10.348768 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:10.348852 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:10.349019 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 12:25:10.349061 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":73,"threads_started":1}
I20260928 12:25:10.349236 20295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:10.345023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:43477 (local address 127.0.0.1:51140)
0928 12:25:10.345346 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:10.345386 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:10.345453 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:10.345633 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:10.345636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:10.345638 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:25:10.345695 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:10.345696 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:10.346927 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:10.346928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:10.348125 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:10.348127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:10.349024 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:10.349025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:10.349083 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:10.349087 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:25:10.349134 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 12:25:10.349172 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":7,"spinlock_wait_cycles":896,"thread_start_us":133,"threads_started":1}
I20260928 12:25:10.349496 20297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:10.345213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51140 (local address 127.2.56.3:43477)
0928 12:25:10.345374 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:10.345378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:10.345449 (+ 71us) server_negotiation.cc:407] Connection header received
0928 12:25:10.345531 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:10.345533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:10.345547 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:10.345629 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:25:10.345769 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:10.346841 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:10.348210 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:10.348866 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:10.348921 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:10.348990 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:10.349391 (+ 401us) server_negotiation.cc:299] Negotiation successful
0928 12:25:10.349428 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":95,"threads_started":1}
I20260928 12:25:10.349517 20300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:10.345024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:37253 (local address 127.0.0.1:56016)
0928 12:25:10.345581 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:10.345591 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:10.345674 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:10.345930 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:10.345931 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:10.345933 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:25:10.345987 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:10.345989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:10.346831 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:10.346832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:10.348125 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:10.348127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:10.348882 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:10.348883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:10.348936 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:10.348939 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:25:10.349406 (+ 467us) client_negotiation.cc:240] Negotiation successful
0928 12:25:10.349451 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":29,"thread_start_us":520,"threads_started":1}
I20260928 12:25:10.349687 20090 tablet_service.cc:1467] Processing CreateTablet for tablet c8731de0f1094f60b2779fe571482787 (DEFAULT_TABLE table=test-table [id=b8af5bfbbd2e48c78a398ecb77af2633]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:25:10.350057 20090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8731de0f1094f60b2779fe571482787. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:10.350034 20221 tablet_service.cc:1467] Processing CreateTablet for tablet c8731de0f1094f60b2779fe571482787 (DEFAULT_TABLE table=test-table [id=b8af5bfbbd2e48c78a398ecb77af2633]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:25:10.350157 19959 tablet_service.cc:1467] Processing CreateTablet for tablet c8731de0f1094f60b2779fe571482787 (DEFAULT_TABLE table=test-table [id=b8af5bfbbd2e48c78a398ecb77af2633]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:25:10.350319 20221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8731de0f1094f60b2779fe571482787. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:10.350461 19959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8731de0f1094f60b2779fe571482787. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:10.352701 20301 tablet_bootstrap.cc:492] T c8731de0f1094f60b2779fe571482787 P 9e83769fb7d846bf855cea40798ebcdf: Bootstrap starting.
I20260928 12:25:10.353132 20303 tablet_bootstrap.cc:492] T c8731de0f1094f60b2779fe571482787 P 3baf8a3cf0ce42469c4d2d62e7c8a75d: Bootstrap starting.
I20260928 12:25:10.353420 20301 tablet_bootstrap.cc:654] T c8731de0f1094f60b2779fe571482787 P 9e83769fb7d846bf855cea40798ebcdf: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:10.353724 20303 tablet_bootstrap.cc:654] T c8731de0f1094f60b2779fe571482787 P 3baf8a3cf0ce42469c4d2d62e7c8a75d: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:10.353729 20301 log.cc:824] T c8731de0f1094f60b2779fe571482787 P 9e83769fb7d846bf855cea40798ebcdf: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:10.354015 20303 log.cc:824] T c8731de0f1094f60b2779fe571482787 P 3baf8a3cf0ce42469c4d2d62e7c8a75d: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:10.354290 20302 tablet_bootstrap.cc:492] T c8731de0f1094f60b2779fe571482787 P 0095203c0d004c17b2bad8f110984edb: Bootstrap starting.
I20260928 12:25:10.354449 20301 tablet_bootstrap.cc:492] T c8731de0f1094f60b2779fe571482787 P 9e83769fb7d846bf855cea40798ebcdf: No bootstrap required, opened a new log
I20260928 12:25:10.354516 20301 ts_tablet_manager.cc:1397] T c8731de0f1094f60b2779fe571482787 P 9e83769fb7d846bf855cea40798ebcdf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:25:10.354604 20303 tablet_bootstrap.cc:492] T c8731de0f1094f60b2779fe571482787 P 3baf8a3cf0ce42469c4d2d62e7c8a75d: No bootstrap required, opened a new log
I20260928 12:25:10.354660 20303 ts_tablet_manager.cc:1397] T c8731de0f1094f60b2779fe571482787 P 3baf8a3cf0ce42469c4d2d62e7c8a75d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:25:10.354868 20302 tablet_bootstrap.cc:654] T c8731de0f1094f60b2779fe571482787 P 0095203c0d004c17b2bad8f110984edb: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:10.355177 20302 log.cc:824] T c8731de0f1094f60b2779fe571482787 P 0095203c0d004c17b2bad8f110984edb: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:10.355909 20302 tablet_bootstrap.cc:492] T c8731de0f1094f60b2779fe571482787 P 0095203c0d004c17b2bad8f110984edb: No bootstrap required, opened a new log
I20260928 12:25:10.355968 20302 ts_tablet_manager.cc:1397] T c8731de0f1094f60b2779fe571482787 P 0095203c0d004c17b2bad8f110984edb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:25:10.356720 20303 raft_consensus.cc:348] T c8731de0f1094f60b2779fe571482787 P 3baf8a3cf0ce42469c4d2d62e7c8a75d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0095203c0d004c17b2bad8f110984edb" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35017 } } peers { permanent_uuid: "3baf8a3cf0ce42469c4d2d62e7c8a75d" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 43477 } } peers { permanent_uuid: "9e83769fb7d846bf855cea40798ebcdf" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37253 } }
I20260928 12:25:10.356844 20303 raft_consensus.cc:374] T c8731de0f1094f60b2779fe571482787 P 3baf8a3cf0ce42469c4d2d62e7c8a75d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:10.356876 20303 raft_consensus.cc:729] T c8731de0f1094f60b2779fe571482787 P 3baf8a3cf0ce42469c4d2d62e7c8a75d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3baf8a3cf0ce42469c4d2d62e7c8a75d, State: Initialized, Role: FOLLOWER
I20260928 12:25:10.356839 20301 raft_consensus.cc:348] T c8731de0f1094f60b2779fe571482787 P 9e83769fb7d846bf855cea40798ebcdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0095203c0d004c17b2bad8f110984edb" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35017 } } peers { permanent_uuid: "3baf8a3cf0ce42469c4d2d62e7c8a75d" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 43477 } } peers { permanent_uuid: "9e83769fb7d846bf855cea40798ebcdf" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37253 } }
I20260928 12:25:10.356985 20303 consensus_queue.cc:260] T c8731de0f1094f60b2779fe571482787 P 3baf8a3cf0ce42469c4d2d62e7c8a75d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0095203c0d004c17b2bad8f110984edb" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35017 } } peers { permanent_uuid: "3baf8a3cf0ce42469c4d2d62e7c8a75d" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 43477 } } peers { permanent_uuid: "9e83769fb7d846bf855cea40798ebcdf" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37253 } }
I20260928 12:25:10.357025 20301 raft_consensus.cc:374] T c8731de0f1094f60b2779fe571482787 P 9e83769fb7d846bf855cea40798ebcdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:10.357062 20301 raft_consensus.cc:729] T c8731de0f1094f60b2779fe571482787 P 9e83769fb7d846bf855cea40798ebcdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e83769fb7d846bf855cea40798ebcdf, State: Initialized, Role: FOLLOWER
I20260928 12:25:10.357187 20301 consensus_queue.cc:260] T c8731de0f1094f60b2779fe571482787 P 9e83769fb7d846bf855cea40798ebcdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0095203c0d004c17b2bad8f110984edb" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35017 } } peers { permanent_uuid: "3baf8a3cf0ce42469c4d2d62e7c8a75d" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 43477 } } peers { permanent_uuid: "9e83769fb7d846bf855cea40798ebcdf" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37253 } }
I20260928 12:25:10.357234 20303 ts_tablet_manager.cc:1428] T c8731de0f1094f60b2779fe571482787 P 3baf8a3cf0ce42469c4d2d62e7c8a75d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:25:10.357369 20287 heartbeater.cc:503] Master 127.2.56.62:44791 was elected leader, sending a full tablet report...
I20260928 12:25:10.357458 20301 ts_tablet_manager.cc:1428] T c8731de0f1094f60b2779fe571482787 P 9e83769fb7d846bf855cea40798ebcdf: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260928 12:25:10.357615 20025 heartbeater.cc:503] Master 127.2.56.62:44791 was elected leader, sending a full tablet report...
I20260928 12:25:10.358297 20302 raft_consensus.cc:348] T c8731de0f1094f60b2779fe571482787 P 0095203c0d004c17b2bad8f110984edb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0095203c0d004c17b2bad8f110984edb" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35017 } } peers { permanent_uuid: "3baf8a3cf0ce42469c4d2d62e7c8a75d" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 43477 } } peers { permanent_uuid: "9e83769fb7d846bf855cea40798ebcdf" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37253 } }
I20260928 12:25:10.358440 20302 raft_consensus.cc:374] T c8731de0f1094f60b2779fe571482787 P 0095203c0d004c17b2bad8f110984edb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:10.358481 20302 raft_consensus.cc:729] T c8731de0f1094f60b2779fe571482787 P 0095203c0d004c17b2bad8f110984edb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0095203c0d004c17b2bad8f110984edb, State: Initialized, Role: FOLLOWER
I20260928 12:25:10.358572 20302 consensus_queue.cc:260] T c8731de0f1094f60b2779fe571482787 P 0095203c0d004c17b2bad8f110984edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0095203c0d004c17b2bad8f110984edb" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35017 } } peers { permanent_uuid: "3baf8a3cf0ce42469c4d2d62e7c8a75d" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 43477 } } peers { permanent_uuid: "9e83769fb7d846bf855cea40798ebcdf" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37253 } }
I20260928 12:25:10.358741 20302 ts_tablet_manager.cc:1428] T c8731de0f1094f60b2779fe571482787 P 0095203c0d004c17b2bad8f110984edb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:25:10.358798 20156 heartbeater.cc:503] Master 127.2.56.62:44791 was elected leader, sending a full tablet report...
I20260928 12:25:10.405156 20309 raft_consensus.cc:482] T c8731de0f1094f60b2779fe571482787 P 0095203c0d004c17b2bad8f110984edb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:25:10.405304 20309 raft_consensus.cc:504] T c8731de0f1094f60b2779fe571482787 P 0095203c0d004c17b2bad8f110984edb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0095203c0d004c17b2bad8f110984edb" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35017 } } peers { permanent_uuid: "3baf8a3cf0ce42469c4d2d62e7c8a75d" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 43477 } } peers { permanent_uuid: "9e83769fb7d846bf855cea40798ebcdf" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37253 } }
I20260928 12:25:10.405634 20309 leader_election.cc:288] T c8731de0f1094f60b2779fe571482787 P 0095203c0d004c17b2bad8f110984edb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3baf8a3cf0ce42469c4d2d62e7c8a75d (127.2.56.3:43477), 9e83769fb7d846bf855cea40798ebcdf (127.2.56.1:37253)
I20260928 12:25:10.409533 20158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:10.405813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:37253 (local address 127.2.56.2:41179)
0928 12:25:10.405866 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:10.405880 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:10.405928 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:10.406141 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:10.406142 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:10.406143 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:25:10.406210 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:10.406211 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:10.407140 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:10.407141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:10.408486 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:10.408489 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:10.409209 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:10.409210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:10.409336 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:10.409342 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 12:25:10.409399 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 12:25:10.409440 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":64}
I20260928 12:25:10.409566 20299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:10.405935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:41179 (local address 127.2.56.1:37253)
0928 12:25:10.405986 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:10.405988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:10.405996 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:25:10.406041 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:10.406043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:10.406052 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:10.406128 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:25:10.406280 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:10.407023 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:10.408584 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:10.409145 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:10.409201 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:10.409210 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:10.409443 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 12:25:10.409489 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 12:25:10.409557 20297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:10.405902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:59523 (local address 127.2.56.3:43477)
0928 12:25:10.405959 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:10.405961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:10.406096 (+ 135us) server_negotiation.cc:407] Connection header received
0928 12:25:10.406128 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:10.406128 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:10.406137 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:10.406212 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:25:10.406291 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:10.407047 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:10.408584 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:10.409124 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:10.409195 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:10.409206 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:10.409443 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 12:25:10.409489 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 12:25:10.409538 20310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:10.405813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:43477 (local address 127.2.56.2:59523)
0928 12:25:10.406028 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:10.406037 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:10.406087 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:10.406193 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:10.406194 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:10.406195 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:25:10.406244 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:10.406245 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:10.407140 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:10.407141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:10.408486 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:10.408489 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:10.409210 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:10.409211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:10.409336 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:10.409342 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 12:25:10.409399 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 12:25:10.409440 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":62,"thread_start_us":104,"threads_started":1}
I20260928 12:25:10.409878 19979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8731de0f1094f60b2779fe571482787" candidate_uuid: "0095203c0d004c17b2bad8f110984edb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e83769fb7d846bf855cea40798ebcdf" is_pre_election: true
I20260928 12:25:10.410029 19979 raft_consensus.cc:2454] T c8731de0f1094f60b2779fe571482787 P 9e83769fb7d846bf855cea40798ebcdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0095203c0d004c17b2bad8f110984edb in term 0.
I20260928 12:25:10.410239 20044 leader_election.cc:302] T c8731de0f1094f60b2779fe571482787 P 0095203c0d004c17b2bad8f110984edb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0095203c0d004c17b2bad8f110984edb, 9e83769fb7d846bf855cea40798ebcdf; no voters:
I20260928 12:25:10.410351 20309 raft_consensus.cc:2781] T c8731de0f1094f60b2779fe571482787 P 0095203c0d004c17b2bad8f110984edb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:25:10.410338 20241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8731de0f1094f60b2779fe571482787" candidate_uuid: "0095203c0d004c17b2bad8f110984edb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3baf8a3cf0ce42469c4d2d62e7c8a75d" is_pre_election: true
I20260928 12:25:10.410398 20309 raft_consensus.cc:482] T c8731de0f1094f60b2779fe571482787 P 0095203c0d004c17b2bad8f110984edb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:25:10.410413 20309 raft_consensus.cc:3037] T c8731de0f1094f60b2779fe571482787 P 0095203c0d004c17b2bad8f110984edb [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:10.410465 20241 raft_consensus.cc:2454] T c8731de0f1094f60b2779fe571482787 P 3baf8a3cf0ce42469c4d2d62e7c8a75d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0095203c0d004c17b2bad8f110984edb in term 0.
I20260928 12:25:10.411096 20309 raft_consensus.cc:504] T c8731de0f1094f60b2779fe571482787 P 0095203c0d004c17b2bad8f110984edb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0095203c0d004c17b2bad8f110984edb" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35017 } } peers { permanent_uuid: "3baf8a3cf0ce42469c4d2d62e7c8a75d" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 43477 } } peers { permanent_uuid: "9e83769fb7d846bf855cea40798ebcdf" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37253 } }
I20260928 12:25:10.411262 20309 leader_election.cc:288] T c8731de0f1094f60b2779fe571482787 P 0095203c0d004c17b2bad8f110984edb [CANDIDATE]: Term 1 election: Requested vote from peers 3baf8a3cf0ce42469c4d2d62e7c8a75d (127.2.56.3:43477), 9e83769fb7d846bf855cea40798ebcdf (127.2.56.1:37253)
I20260928 12:25:10.411327 19979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8731de0f1094f60b2779fe571482787" candidate_uuid: "0095203c0d004c17b2bad8f110984edb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e83769fb7d846bf855cea40798ebcdf"
I20260928 12:25:10.411408 19979 raft_consensus.cc:3037] T c8731de0f1094f60b2779fe571482787 P 9e83769fb7d846bf855cea40798ebcdf [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:10.412325 19979 raft_consensus.cc:2454] T c8731de0f1094f60b2779fe571482787 P 9e83769fb7d846bf855cea40798ebcdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0095203c0d004c17b2bad8f110984edb in term 1.
I20260928 12:25:10.412444 20241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8731de0f1094f60b2779fe571482787" candidate_uuid: "0095203c0d004c17b2bad8f110984edb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3baf8a3cf0ce42469c4d2d62e7c8a75d"
I20260928 12:25:10.412467 20044 leader_election.cc:302] T c8731de0f1094f60b2779fe571482787 P 0095203c0d004c17b2bad8f110984edb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0095203c0d004c17b2bad8f110984edb, 9e83769fb7d846bf855cea40798ebcdf; no voters:
I20260928 12:25:10.412536 20241 raft_consensus.cc:3037] T c8731de0f1094f60b2779fe571482787 P 3baf8a3cf0ce42469c4d2d62e7c8a75d [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:10.412606 20309 raft_consensus.cc:2781] T c8731de0f1094f60b2779fe571482787 P 0095203c0d004c17b2bad8f110984edb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:25:10.412798 20309 raft_consensus.cc:686] T c8731de0f1094f60b2779fe571482787 P 0095203c0d004c17b2bad8f110984edb [term 1 LEADER]: Becoming Leader. State: Replica: 0095203c0d004c17b2bad8f110984edb, State: Running, Role: LEADER
I20260928 12:25:10.412972 20309 consensus_queue.cc:237] T c8731de0f1094f60b2779fe571482787 P 0095203c0d004c17b2bad8f110984edb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0095203c0d004c17b2bad8f110984edb" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35017 } } peers { permanent_uuid: "3baf8a3cf0ce42469c4d2d62e7c8a75d" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 43477 } } peers { permanent_uuid: "9e83769fb7d846bf855cea40798ebcdf" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37253 } }
I20260928 12:25:10.413380 20241 raft_consensus.cc:2454] T c8731de0f1094f60b2779fe571482787 P 3baf8a3cf0ce42469c4d2d62e7c8a75d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0095203c0d004c17b2bad8f110984edb in term 1.
I20260928 12:25:10.413828 19837 catalog_manager.cc:5495] T c8731de0f1094f60b2779fe571482787 P 0095203c0d004c17b2bad8f110984edb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0095203c0d004c17b2bad8f110984edb (127.2.56.2). New cstate: current_term: 1 leader_uuid: "0095203c0d004c17b2bad8f110984edb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0095203c0d004c17b2bad8f110984edb" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 35017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3baf8a3cf0ce42469c4d2d62e7c8a75d" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 43477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e83769fb7d846bf855cea40798ebcdf" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37253 } health_report { overall_health: UNKNOWN } } }
I20260928 12:25:10.424261 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:10.422098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56596 (local address 127.2.56.2:35017)
0928 12:25:10.422151 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:10.422152 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:10.422160 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:25:10.422214 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:10.422215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:10.422231 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:10.422285 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:25:10.422367 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:10.422835 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:10.423548 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:10.423732 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:10.423784 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:10.423973 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:25:10.424073 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:25:10.424134 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:10.424149 (+ 15us) server_negotiation.cc:299] Negotiation successful
0928 12:25:10.424165 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 12:25:10.425920 19979 raft_consensus.cc:1261] T c8731de0f1094f60b2779fe571482787 P 9e83769fb7d846bf855cea40798ebcdf [term 1 FOLLOWER]: Refusing update from remote peer 0095203c0d004c17b2bad8f110984edb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:25:10.425925 20241 raft_consensus.cc:1261] T c8731de0f1094f60b2779fe571482787 P 3baf8a3cf0ce42469c4d2d62e7c8a75d [term 1 FOLLOWER]: Refusing update from remote peer 0095203c0d004c17b2bad8f110984edb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:25:10.426255 20311 consensus_queue.cc:1035] T c8731de0f1094f60b2779fe571482787 P 0095203c0d004c17b2bad8f110984edb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3baf8a3cf0ce42469c4d2d62e7c8a75d" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 43477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:25:10.426374 20309 consensus_queue.cc:1035] T c8731de0f1094f60b2779fe571482787 P 0095203c0d004c17b2bad8f110984edb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e83769fb7d846bf855cea40798ebcdf" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:25:10.428544 20314 mvcc.cc:205] Tried to move back new op lower bound from 7334290679502290944 to 7334290679452368896. Current Snapshot: MvccSnapshot[applied={T|T < 7334290679502290944}]
I20260928 12:25:10.429005 20316 mvcc.cc:205] Tried to move back new op lower bound from 7334290679502290944 to 7334290679452368896. Current Snapshot: MvccSnapshot[applied={T|T < 7334290679502290944 or (T in {7334290679502290944})}]
I20260928 12:25:10.429114 20315 mvcc.cc:205] Tried to move back new op lower bound from 7334290679502290944 to 7334290679452368896. Current Snapshot: MvccSnapshot[applied={T|T < 7334290679502290944}]
I20260928 12:25:10.431007 19836 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:25:10.431090 19836 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:25:10.431704 19836 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 46421a3cfca94cc4a42c5a33caddb729: Sending DeleteTablet for 3 replicas of tablet c8731de0f1094f60b2779fe571482787
I20260928 12:25:10.431967 20090 tablet_service.cc:1514] Processing DeleteTablet for tablet c8731de0f1094f60b2779fe571482787 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:25:10 UTC) from {username='slave'} at 127.0.0.1:56594
I20260928 12:25:10.432073 20221 tablet_service.cc:1514] Processing DeleteTablet for tablet c8731de0f1094f60b2779fe571482787 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:25:10 UTC) from {username='slave'} at 127.0.0.1:51140
I20260928 12:25:10.432173 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 19897
I20260928 12:25:10.432261 19959 tablet_service.cc:1514] Processing DeleteTablet for tablet c8731de0f1094f60b2779fe571482787 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:25:10 UTC) from {username='slave'} at 127.0.0.1:56016
I20260928 12:25:10.432648 20324 tablet_replica.cc:330] T c8731de0f1094f60b2779fe571482787 P 0095203c0d004c17b2bad8f110984edb: stopping tablet replica
I20260928 12:25:10.432737 20324 raft_consensus.cc:2229] T c8731de0f1094f60b2779fe571482787 P 0095203c0d004c17b2bad8f110984edb [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:25:10.432852 20324 raft_consensus.cc:2258] T c8731de0f1094f60b2779fe571482787 P 0095203c0d004c17b2bad8f110984edb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:25:10.433213 20324 ts_tablet_manager.cc:1905] T c8731de0f1094f60b2779fe571482787 P 0095203c0d004c17b2bad8f110984edb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:25:10.434774 20324 ts_tablet_manager.cc:1918] T c8731de0f1094f60b2779fe571482787 P 0095203c0d004c17b2bad8f110984edb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:25:10.434836 20324 log.cc:1196] T c8731de0f1094f60b2779fe571482787 P 0095203c0d004c17b2bad8f110984edb: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-1/wal/wals/c8731de0f1094f60b2779fe571482787
I20260928 12:25:10.435070 20324 ts_tablet_manager.cc:1939] T c8731de0f1094f60b2779fe571482787 P 0095203c0d004c17b2bad8f110984edb: Deleting consensus metadata
I20260928 12:25:10.435518 19821 catalog_manager.cc:4841] TS 0095203c0d004c17b2bad8f110984edb (127.2.56.2:35017): tablet c8731de0f1094f60b2779fe571482787 (table test-table [id=b8af5bfbbd2e48c78a398ecb77af2633]) successfully deleted
I20260928 12:25:10.437378 20325 tablet_replica.cc:330] T c8731de0f1094f60b2779fe571482787 P 3baf8a3cf0ce42469c4d2d62e7c8a75d: stopping tablet replica
I20260928 12:25:10.437484 20325 raft_consensus.cc:2229] T c8731de0f1094f60b2779fe571482787 P 3baf8a3cf0ce42469c4d2d62e7c8a75d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:25:10.437602 20325 raft_consensus.cc:2258] T c8731de0f1094f60b2779fe571482787 P 3baf8a3cf0ce42469c4d2d62e7c8a75d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:25:10.437959 20325 ts_tablet_manager.cc:1905] T c8731de0f1094f60b2779fe571482787 P 3baf8a3cf0ce42469c4d2d62e7c8a75d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 12:25:10.439255 19822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.56.1:37253 (error 108)
I20260928 12:25:10.439476 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 20028
I20260928 12:25:10.439566 20325 ts_tablet_manager.cc:1918] T c8731de0f1094f60b2779fe571482787 P 3baf8a3cf0ce42469c4d2d62e7c8a75d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:25:10.439625 20325 log.cc:1196] T c8731de0f1094f60b2779fe571482787 P 3baf8a3cf0ce42469c4d2d62e7c8a75d: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598072638673-2272-0/minicluster-data/ts-2/wal/wals/c8731de0f1094f60b2779fe571482787
I20260928 12:25:10.439710 20295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:10.439541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:37253 (local address 127.0.0.1:56030)
0928 12:25:10.439636 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:10.439677 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.1:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 12:25:10.439857 20325 ts_tablet_manager.cc:1939] T c8731de0f1094f60b2779fe571482787 P 3baf8a3cf0ce42469c4d2d62e7c8a75d: Deleting consensus metadata
W20260928 12:25:10.440320 19822 catalog_manager.cc:4584] TS 9e83769fb7d846bf855cea40798ebcdf (127.2.56.1:37253): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c8731de0f1094f60b2779fe571482787: Network error: Client connection negotiation failed: client connection to 127.2.56.1:37253: connect: Connection refused (error 111)
I20260928 12:25:10.440393 19823 catalog_manager.cc:4841] TS 3baf8a3cf0ce42469c4d2d62e7c8a75d (127.2.56.3:43477): tablet c8731de0f1094f60b2779fe571482787 (table test-table [id=b8af5bfbbd2e48c78a398ecb77af2633]) successfully deleted
I20260928 12:25:10.444523 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 20159
I20260928 12:25:10.448666 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 19806
2026-09-28T12:25:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (653 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:25:10 dist-test-slave-920p krb5kdc[20329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:25:10 dist-test-slave-920p krb5kdc[20329](info): set up 2 sockets
Sep 28 12:25:10 dist-test-slave-920p krb5kdc[20329](info): commencing operation
krb5kdc: starting...
W20260928 12:25:11.401360 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:25:11 dist-test-slave-920p krb5kdc[20329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:25:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:25:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
I20260928 12:25:11.513523 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:33349
--webserver_interface=127.2.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:35655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:33349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:25:11.600445 20342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:11.600659 20342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:11.600689 20342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:11.601971 20342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:25:11.602015 20342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:11.602028 20342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:11.602039 20342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:25:11.602056 20342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:25:11.603571 20342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:35655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:33349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:33349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:11.603812 20342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:11.604040 20342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:11.609365 20347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:11.609481 20342 server_base.cc:1034] running on GCE node
W20260928 12:25:11.609495 20348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:11.609372 20350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:11.609988 20342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:11.610361 20342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:11.611563 20342 hybrid_clock.cc:648] HybridClock initialized: now 1790598311611536 us; error 43 us; skew 500 ppm
Sep 28 12:25:11 dist-test-slave-920p krb5kdc[20329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598311, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:11.615065 20342 init.cc:377] Logged in from keytab as kudu/127.2.56.62@KRBTEST.COM (short username kudu)
I20260928 12:25:11.615660 20342 webserver.cc:460] Webserver started at http://127.2.56.62:43029/ using document root <none> and password file <none>
I20260928 12:25:11.615932 20342 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:11.615998 20342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:11.616140 20342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:11.617627 20342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "6ca906222185497aab5ce1cfea75a2cd"
format_stamp: "Formatted at 2026-09-28 12:25:11 on dist-test-slave-920p"
server_key: "4e3dc14e4c85180f82acebed40a36d50"
server_key_iv: "f221c1a238499abef8d57bae20f7e97a"
server_key_version: "encryptionkey@0"
I20260928 12:25:11.618052 20342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "6ca906222185497aab5ce1cfea75a2cd"
format_stamp: "Formatted at 2026-09-28 12:25:11 on dist-test-slave-920p"
server_key: "4e3dc14e4c85180f82acebed40a36d50"
server_key_iv: "f221c1a238499abef8d57bae20f7e97a"
server_key_version: "encryptionkey@0"
I20260928 12:25:11.619946 20342 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 12:25:11.620694 20357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:11.620867 20342 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:25:11.620957 20342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "6ca906222185497aab5ce1cfea75a2cd"
format_stamp: "Formatted at 2026-09-28 12:25:11 on dist-test-slave-920p"
server_key: "4e3dc14e4c85180f82acebed40a36d50"
server_key_iv: "f221c1a238499abef8d57bae20f7e97a"
server_key_version: "encryptionkey@0"
I20260928 12:25:11.621061 20342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:11.645642 20342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:11.647392 20342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:11.647502 20342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:11.651298 20342 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:33349
I20260928 12:25:11.651346 20409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:33349 every 8 connection(s)
I20260928 12:25:11.651664 20342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:25:11.652529 20410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:11.654868 20410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ca906222185497aab5ce1cfea75a2cd: Bootstrap starting.
I20260928 12:25:11.655409 20410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ca906222185497aab5ce1cfea75a2cd: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:11.655654 20410 log.cc:824] T 00000000000000000000000000000000 P 6ca906222185497aab5ce1cfea75a2cd: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:11.656313 20410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ca906222185497aab5ce1cfea75a2cd: No bootstrap required, opened a new log
I20260928 12:25:11.658105 20410 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ca906222185497aab5ce1cfea75a2cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca906222185497aab5ce1cfea75a2cd" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 33349 } }
I20260928 12:25:11.658195 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 20342
I20260928 12:25:11.658249 20410 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ca906222185497aab5ce1cfea75a2cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:11.658285 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:25:11.658283 20410 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ca906222185497aab5ce1cfea75a2cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ca906222185497aab5ce1cfea75a2cd, State: Initialized, Role: FOLLOWER
I20260928 12:25:11.658420 20410 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ca906222185497aab5ce1cfea75a2cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca906222185497aab5ce1cfea75a2cd" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 33349 } }
I20260928 12:25:11.658488 2272 external_mini_cluster.cc:1428] Setting key 6417eb6466af3225a886c1c76a89477a
I20260928 12:25:11.658497 20410 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ca906222185497aab5ce1cfea75a2cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:25:11.658530 20410 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ca906222185497aab5ce1cfea75a2cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:25:11.658586 20410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ca906222185497aab5ce1cfea75a2cd [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:11.659636 20410 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ca906222185497aab5ce1cfea75a2cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca906222185497aab5ce1cfea75a2cd" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 33349 } }
I20260928 12:25:11.659744 20410 leader_election.cc:302] T 00000000000000000000000000000000 P 6ca906222185497aab5ce1cfea75a2cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ca906222185497aab5ce1cfea75a2cd; no voters:
I20260928 12:25:11.659884 20410 leader_election.cc:288] T 00000000000000000000000000000000 P 6ca906222185497aab5ce1cfea75a2cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:25:11.659905 20415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ca906222185497aab5ce1cfea75a2cd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:25:11.660008 20415 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ca906222185497aab5ce1cfea75a2cd [term 1 LEADER]: Becoming Leader. State: Replica: 6ca906222185497aab5ce1cfea75a2cd, State: Running, Role: LEADER
I20260928 12:25:11.660133 20410 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ca906222185497aab5ce1cfea75a2cd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:25:11.660115 20415 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ca906222185497aab5ce1cfea75a2cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca906222185497aab5ce1cfea75a2cd" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 33349 } }
I20260928 12:25:11.660427 20415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ca906222185497aab5ce1cfea75a2cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ca906222185497aab5ce1cfea75a2cd. Latest consensus state: current_term: 1 leader_uuid: "6ca906222185497aab5ce1cfea75a2cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca906222185497aab5ce1cfea75a2cd" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 33349 } } }
I20260928 12:25:11.660512 20415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ca906222185497aab5ce1cfea75a2cd [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:11.660440 20416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ca906222185497aab5ce1cfea75a2cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ca906222185497aab5ce1cfea75a2cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca906222185497aab5ce1cfea75a2cd" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 33349 } } }
I20260928 12:25:11.660735 20416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ca906222185497aab5ce1cfea75a2cd [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:11.660748 20421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:25:11.661372 20421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:25:11.663463 20421 catalog_manager.cc:1347] Generated new cluster ID: c528adee672843ddaf349e70430374b3
I20260928 12:25:11.663515 20421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:25:11 dist-test-slave-920p krb5kdc[20329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:25:11.670199 20414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:11.659013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46424 (local address 127.2.56.62:33349)
0928 12:25:11.659254 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:11.659264 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:11.659301 (+ 37us) server_negotiation.cc:407] Connection header received
0928 12:25:11.659420 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:11.659427 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:11.659481 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:11.659624 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:11.660306 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:11.661747 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:11.662546 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:11.663127 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:11.663231 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:11.667339 (+ 4108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:11.667343 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:11.667344 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:11.667356 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:11.668813 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:11.669119 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:11.669123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:11.669123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:11.669212 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:11.669404 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:11.669405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:11.669406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:11.669633 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:11.669781 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:11.669870 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 12:25:11.669998 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":136,"threads_started":1}
I20260928 12:25:11.702350 20421 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:25:11.703074 20421 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:25:11.713361 20421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ca906222185497aab5ce1cfea75a2cd: Generated new TSK 0
I20260928 12:25:11.713549 20421 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab.
Entry for principal kudu/127.2.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab.
I20260928 12:25:11.752573 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.1:0
--local_ip_for_outbound_sockets=127.2.56.1
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:33349
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:35655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:25:11.833061 20436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:11.833299 20436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:11.833333 20436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:11.834604 20436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:11.834654 20436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:11.834681 20436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.1
I20260928 12:25:11.836195 20436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:35655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:33349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.1
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:11.836416 20436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:11.836637 20436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:11.841439 20441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:11.841467 20442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:11.841439 20444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:11.841454 20436 server_base.cc:1034] running on GCE node
I20260928 12:25:11.841821 20436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:11.842067 20436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:11.843200 20436 hybrid_clock.cc:648] HybridClock initialized: now 1790598311843192 us; error 28 us; skew 500 ppm
Sep 28 12:25:11 dist-test-slave-920p krb5kdc[20329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598311, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:11.845680 20436 init.cc:377] Logged in from keytab as kudu/127.2.56.1@KRBTEST.COM (short username kudu)
I20260928 12:25:11.846088 20436 webserver.cc:460] Webserver started at http://127.2.56.1:37851/ using document root <none> and password file <none>
I20260928 12:25:11.846278 20436 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:11.846318 20436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:11.846407 20436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:11.847402 20436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-0/data/instance:
uuid: "385640e696a74e0f91a73776a38fff68"
format_stamp: "Formatted at 2026-09-28 12:25:11 on dist-test-slave-920p"
server_key: "04d238d85c228367cdfc4dc90ea4ba6c"
server_key_iv: "fe8b5ca7178d53d9698799a9ccd164ab"
server_key_version: "encryptionkey@0"
I20260928 12:25:11.847680 20436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance:
uuid: "385640e696a74e0f91a73776a38fff68"
format_stamp: "Formatted at 2026-09-28 12:25:11 on dist-test-slave-920p"
server_key: "04d238d85c228367cdfc4dc90ea4ba6c"
server_key_iv: "fe8b5ca7178d53d9698799a9ccd164ab"
server_key_version: "encryptionkey@0"
I20260928 12:25:11.848948 20436 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:25:11.849669 20451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:11.849825 20436 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:11.849901 20436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-0/wal
uuid: "385640e696a74e0f91a73776a38fff68"
format_stamp: "Formatted at 2026-09-28 12:25:11 on dist-test-slave-920p"
server_key: "04d238d85c228367cdfc4dc90ea4ba6c"
server_key_iv: "fe8b5ca7178d53d9698799a9ccd164ab"
server_key_version: "encryptionkey@0"
I20260928 12:25:11.849954 20436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:11.864862 20436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:11.866788 20436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:11.866899 20436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:11.867077 20436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:25:11.867340 20436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:25:11.867372 20436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:11.867398 20436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:25:11.867414 20436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:11.872874 20436 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.1:41277
I20260928 12:25:11.872915 20564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.1:41277 every 8 connection(s)
I20260928 12:25:11.873245 20436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
I20260928 12:25:11.877100 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 20436
I20260928 12:25:11.877210 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance
I20260928 12:25:11.877452 2272 external_mini_cluster.cc:1428] Setting key 2ef812f27608a94de7d667e3248e9046
Sep 28 12:25:11 dist-test-slave-920p krb5kdc[20329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598311, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:25:11.884073 20414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:11.873970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:53761 (local address 127.2.56.62:33349)
0928 12:25:11.874024 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:11.874026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:11.874211 (+ 185us) server_negotiation.cc:407] Connection header received
0928 12:25:11.874686 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:11.874688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:11.874700 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:11.874749 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:11.875695 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:11.876950 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:11.878345 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:11.878608 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:11.878677 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:11.881163 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:11.881165 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:11.881166 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:11.881174 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:11.882614 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:11.882991 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:11.882992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:11.882993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:11.883056 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:11.883336 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:11.883337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:11.883337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:11.883483 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:11.883557 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:11.883880 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 12:25:11.883969 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 12:25:11.884532 20567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:11.873940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:33349 (local address 127.2.56.1:53761)
0928 12:25:11.874094 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:11.874129 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:11.874604 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:11.874807 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:11.874811 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:11.875380 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:11.875581 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:11.875585 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:11.877136 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:11.877137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:11.878258 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:11.878262 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:11.878713 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:11.878714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:11.878862 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:11.879566 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:25:11.879576 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:11.881042 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:11.882690 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:11.882697 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:11.882699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:11.882911 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:11.883147 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:11.883148 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:11.883148 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:11.883264 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:11.883566 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:11.883572 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:11.883738 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 12:25:11.883845 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 12:25:11.884083 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
I20260928 12:25:11.885102 20565 heartbeater.cc:347] Connected to a master server at 127.2.56.62:33349
I20260928 12:25:11.885214 20565 heartbeater.cc:464] Registering TS with master...
I20260928 12:25:11.885458 20565 heartbeater.cc:511] Master 127.2.56.62:33349 requested a full tablet report, sending...
I20260928 12:25:11.886065 20374 ts_manager.cc:194] Registered new tserver with Master: 385640e696a74e0f91a73776a38fff68 (127.2.56.1:41277)
WARNING: no policy specified for kudu/127.2.56.2@KRBTEST.COM; defaulting to no policy
I20260928 12:25:11.886987 20374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.1@KRBTEST.COM'} at 127.2.56.1:53761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab.
Entry for principal kudu/127.2.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab.
I20260928 12:25:11.904582 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.2:0
--local_ip_for_outbound_sockets=127.2.56.2
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:33349
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:35655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:25:11.985596 20570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:11.985800 20570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:11.985821 20570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:11.986922 20570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:11.986954 20570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:11.986979 20570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.2
I20260928 12:25:11.988332 20570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:35655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:33349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.2
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:11.988507 20570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:11.988696 20570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:11.993839 20575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:11.993919 20570 server_base.cc:1034] running on GCE node
W20260928 12:25:11.993839 20578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:11.993834 20576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:11.994270 20570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:11.994510 20570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:11.995683 20570 hybrid_clock.cc:648] HybridClock initialized: now 1790598311995659 us; error 28 us; skew 500 ppm
Sep 28 12:25:11 dist-test-slave-920p krb5kdc[20329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598311, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:11.998152 20570 init.cc:377] Logged in from keytab as kudu/127.2.56.2@KRBTEST.COM (short username kudu)
I20260928 12:25:11.998582 20570 webserver.cc:460] Webserver started at http://127.2.56.2:44559/ using document root <none> and password file <none>
I20260928 12:25:11.998764 20570 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:11.998813 20570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:11.998919 20570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:11.999899 20570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-1/data/instance:
uuid: "84446b39131540478e4d28ca20bccaf3"
format_stamp: "Formatted at 2026-09-28 12:25:11 on dist-test-slave-920p"
server_key: "d402fdc94d8d8ced360352409502d03f"
server_key_iv: "aaedfb888a2ed9e5178cb6eacb9b456e"
server_key_version: "encryptionkey@0"
I20260928 12:25:12.000196 20570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance:
uuid: "84446b39131540478e4d28ca20bccaf3"
format_stamp: "Formatted at 2026-09-28 12:25:11 on dist-test-slave-920p"
server_key: "d402fdc94d8d8ced360352409502d03f"
server_key_iv: "aaedfb888a2ed9e5178cb6eacb9b456e"
server_key_version: "encryptionkey@0"
I20260928 12:25:12.001408 20570 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 12:25:12.002189 20585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:12.002353 20570 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:25:12.002434 20570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-1/wal
uuid: "84446b39131540478e4d28ca20bccaf3"
format_stamp: "Formatted at 2026-09-28 12:25:11 on dist-test-slave-920p"
server_key: "d402fdc94d8d8ced360352409502d03f"
server_key_iv: "aaedfb888a2ed9e5178cb6eacb9b456e"
server_key_version: "encryptionkey@0"
I20260928 12:25:12.002489 20570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:12.024933 20570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:12.027073 20570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:12.027200 20570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:12.027386 20570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:25:12.027633 20570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:25:12.027666 20570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:12.027696 20570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:25:12.027707 20570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:12.032851 20570 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.2:34759
I20260928 12:25:12.032936 20698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.2:34759 every 8 connection(s)
I20260928 12:25:12.033180 20570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
Sep 28 12:25:12 dist-test-slave-920p krb5kdc[20329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598311, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:25:12.039544 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 20570
I20260928 12:25:12.039626 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance
I20260928 12:25:12.039829 2272 external_mini_cluster.cc:1428] Setting key fe28d7e367a7a6c71c29786abf28fa15
I20260928 12:25:12.042142 20414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:12.033838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:39353 (local address 127.2.56.62:33349)
0928 12:25:12.033882 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:12.033884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:12.034091 (+ 207us) server_negotiation.cc:407] Connection header received
0928 12:25:12.034588 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:12.034589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:12.034602 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:12.034653 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:12.035576 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.036094 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.037065 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.037306 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.037361 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:12.039399 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:12.039401 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:12.039401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:12.039409 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:12.040766 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:12.041209 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:12.041209 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:12.041210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:12.041293 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:12.041521 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:12.041522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:12.041522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:12.041617 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:12.041666 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:12.041964 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 12:25:12.042052 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 12:25:12.042230 20701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:12.033820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:33349 (local address 127.2.56.2:39353)
0928 12:25:12.033951 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:12.033990 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:12.034491 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:12.034720 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:12.034724 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:12.035284 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:12.035484 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:12.035489 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.036180 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.036181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:12.036975 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:12.036977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.037369 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.037370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:12.037493 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:12.037900 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:25:12.037909 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:12.039310 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:12.040840 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:12.040848 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:12.040850 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:12.041130 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:12.041341 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:12.041342 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:12.041342 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:12.041467 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:12.041663 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:12.041668 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:12.041833 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 12:25:12.041938 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 12:25:12.042065 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":61,"threads_started":1}
I20260928 12:25:12.042820 20699 heartbeater.cc:347] Connected to a master server at 127.2.56.62:33349
I20260928 12:25:12.042906 20699 heartbeater.cc:464] Registering TS with master...
I20260928 12:25:12.043069 20699 heartbeater.cc:511] Master 127.2.56.62:33349 requested a full tablet report, sending...
I20260928 12:25:12.043406 20374 ts_manager.cc:194] Registered new tserver with Master: 84446b39131540478e4d28ca20bccaf3 (127.2.56.2:34759)
I20260928 12:25:12.043766 20374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.2@KRBTEST.COM'} at 127.2.56.2:39353
WARNING: no policy specified for kudu/127.2.56.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab.
Entry for principal kudu/127.2.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab.
I20260928 12:25:12.064054 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.3:0
--local_ip_for_outbound_sockets=127.2.56.3
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:33349
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:35655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:25:12.155850 20704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:12.156064 20704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:12.156095 20704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:12.157461 20704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:12.157505 20704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:12.157534 20704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.3
I20260928 12:25:12.159020 20704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:35655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:33349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.3
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:12.159227 20704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:12.159406 20704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:12.164309 20709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:12.164326 20710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:12.164284 20712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:12.164629 20704 server_base.cc:1034] running on GCE node
I20260928 12:25:12.164780 20704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:12.164964 20704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:12.166117 20704 hybrid_clock.cc:648] HybridClock initialized: now 1790598312166100 us; error 28 us; skew 500 ppm
Sep 28 12:25:12 dist-test-slave-920p krb5kdc[20329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598312, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:12.168440 20704 init.cc:377] Logged in from keytab as kudu/127.2.56.3@KRBTEST.COM (short username kudu)
I20260928 12:25:12.168884 20704 webserver.cc:460] Webserver started at http://127.2.56.3:35863/ using document root <none> and password file <none>
I20260928 12:25:12.169094 20704 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:12.169143 20704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:12.169297 20704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:12.170347 20704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-2/data/instance:
uuid: "aaf6099376104ccdaddefc70508adeca"
format_stamp: "Formatted at 2026-09-28 12:25:12 on dist-test-slave-920p"
server_key: "1251132ec77bbdac98622ab4cd2e9e0f"
server_key_iv: "d7226f75f4a501c97b3789501a58bf06"
server_key_version: "encryptionkey@0"
I20260928 12:25:12.170692 20704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance:
uuid: "aaf6099376104ccdaddefc70508adeca"
format_stamp: "Formatted at 2026-09-28 12:25:12 on dist-test-slave-920p"
server_key: "1251132ec77bbdac98622ab4cd2e9e0f"
server_key_iv: "d7226f75f4a501c97b3789501a58bf06"
server_key_version: "encryptionkey@0"
I20260928 12:25:12.172224 20704 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:25:12.173134 20719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:12.173564 20704 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:25:12.173667 20704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-2/wal
uuid: "aaf6099376104ccdaddefc70508adeca"
format_stamp: "Formatted at 2026-09-28 12:25:12 on dist-test-slave-920p"
server_key: "1251132ec77bbdac98622ab4cd2e9e0f"
server_key_iv: "d7226f75f4a501c97b3789501a58bf06"
server_key_version: "encryptionkey@0"
I20260928 12:25:12.173738 20704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:12.185968 20704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:12.187777 20704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:12.187927 20704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:12.188133 20704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:25:12.188423 20704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:25:12.188455 20704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:12.188483 20704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:25:12.188500 20704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:12.193825 20704 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.3:45845
I20260928 12:25:12.193892 20832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.3:45845 every 8 connection(s)
I20260928 12:25:12.194162 20704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
I20260928 12:25:12.199331 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 20704
I20260928 12:25:12.199434 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance
I20260928 12:25:12.199611 2272 external_mini_cluster.cc:1428] Setting key 387b3904ed519786b248009ee704b425
Sep 28 12:25:12 dist-test-slave-920p krb5kdc[20329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598312, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:25:12.202736 20414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:12.194800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:53631 (local address 127.2.56.62:33349)
0928 12:25:12.194862 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:12.194863 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:12.195055 (+ 192us) server_negotiation.cc:407] Connection header received
0928 12:25:12.195559 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:12.195560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:12.195569 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:12.195606 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:12.196466 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.196943 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.197958 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.198152 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.198230 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:12.200486 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:12.200487 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:12.200488 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:12.200494 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:12.201574 (+ 1080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:12.201918 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:12.201919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:12.201920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:12.201971 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:12.202178 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:12.202179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:12.202179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:12.202285 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:12.202337 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:12.202577 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 12:25:12.202649 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 12:25:12.202906 20835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:12.194775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:33349 (local address 127.2.56.3:53631)
0928 12:25:12.194934 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:12.194977 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:12.195465 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:12.195665 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:12.195668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:12.196208 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:12.196402 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:12.196406 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.197003 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.197004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:12.197866 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:12.197869 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.198269 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.198270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:12.198374 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:12.198854 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:25:12.198864 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:12.200407 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:12.201640 (+ 1233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:12.201654 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:12.201656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:12.201856 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:12.202023 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:12.202024 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:12.202024 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:12.202122 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:12.202336 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:12.202342 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:12.202468 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 12:25:12.202551 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 12:25:12.202707 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20260928 12:25:12.203284 20833 heartbeater.cc:347] Connected to a master server at 127.2.56.62:33349
I20260928 12:25:12.203368 20833 heartbeater.cc:464] Registering TS with master...
I20260928 12:25:12.203526 20833 heartbeater.cc:511] Master 127.2.56.62:33349 requested a full tablet report, sending...
I20260928 12:25:12.203804 20374 ts_manager.cc:194] Registered new tserver with Master: aaf6099376104ccdaddefc70508adeca (127.2.56.3:45845)
I20260928 12:25:12.204157 20374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.3@KRBTEST.COM'} at 127.2.56.3:53631
I20260928 12:25:12.211133 2272 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:25:12.217841 20414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:12.212383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46434 (local address 127.2.56.62:33349)
0928 12:25:12.212435 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:12.212437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:12.212522 (+ 85us) server_negotiation.cc:407] Connection header received
0928 12:25:12.212562 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:12.212563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:12.212570 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:12.212612 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:12.213089 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.213612 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.214318 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.214513 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.214574 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:12.215773 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:12.215775 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:12.215775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:12.215781 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:12.216846 (+ 1065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:12.217169 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:12.217170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:12.217171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:12.217224 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:12.217448 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:12.217449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:12.217449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:12.217567 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:12.217643 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:12.217721 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 12:25:12.217764 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 12:25:12.219326 20374 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46434:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:25:12.220927 20374 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:25:12.227008 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:12.222974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:45845 (local address 127.0.0.1:38864)
0928 12:25:12.223142 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:12.223191 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:12.223255 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:12.223532 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:12.223535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:12.223537 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:25:12.223621 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:12.223623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.224913 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.224914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:12.225991 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:12.225992 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.226697 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.226697 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:12.226749 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:12.226760 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 12:25:12.226802 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 12:25:12.226844 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":1,"thread_start_us":58,"threads_started":1}
I20260928 12:25:12.227021 20849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:12.223212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38864 (local address 127.2.56.3:45845)
0928 12:25:12.223368 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:12.223372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:12.223382 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:25:12.223422 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:12.223424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:12.223442 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:12.223520 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:25:12.223702 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.224815 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.226082 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.226542 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.226611 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:12.226666 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:12.226893 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 12:25:12.226933 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":67,"threads_started":1}
I20260928 12:25:12.227175 20845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:12.222979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:41277 (local address 127.0.0.1:52518)
0928 12:25:12.223360 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:12.223368 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:12.223410 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:12.223625 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:12.223626 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:12.223628 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:25:12.223656 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:12.223657 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.224931 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.224932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:12.225756 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:12.225759 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.226607 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.226608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:12.226697 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:12.226711 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 12:25:12.226785 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 12:25:12.226844 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":4,"thread_start_us":87,"threads_started":1}
I20260928 12:25:12.227015 20844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:12.222979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:34759 (local address 127.0.0.1:55482)
0928 12:25:12.223414 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:12.223423 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:12.223478 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:12.223649 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:12.223650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:12.223652 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:25:12.223695 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:12.223697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.224695 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.224696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:12.225964 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:12.225967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.226798 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.226799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:12.226859 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:12.226863 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:25:12.226915 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 12:25:12.226955 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":4,"thread_start_us":84,"threads_started":1}
I20260928 12:25:12.227058 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:12.223164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52518 (local address 127.2.56.1:41277)
0928 12:25:12.223347 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:12.223352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:12.223403 (+ 51us) server_negotiation.cc:407] Connection header received
0928 12:25:12.223470 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:12.223472 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:12.223486 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:12.223551 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:25:12.223734 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.224836 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.225838 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.226520 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.226591 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:12.226646 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:12.226932 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 12:25:12.226973 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":79,"threads_started":1}
I20260928 12:25:12.227705 20848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:12.223171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55482 (local address 127.2.56.2:34759)
0928 12:25:12.223368 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:12.223372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:12.223531 (+ 159us) server_negotiation.cc:407] Connection header received
0928 12:25:12.223570 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:12.223572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:12.223586 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:12.223650 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:25:12.223827 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.224604 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.226050 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.226729 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.226787 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:12.226835 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:12.227051 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 12:25:12.227614 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":87,"threads_started":1}
I20260928 12:25:12.227705 20767 tablet_service.cc:1467] Processing CreateTablet for tablet 8701d169015d41b28db0413d7ebef952 (DEFAULT_TABLE table=test-table [id=1a1ec75518914948a641d0a50e094b3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:25:12.227880 20499 tablet_service.cc:1467] Processing CreateTablet for tablet 8701d169015d41b28db0413d7ebef952 (DEFAULT_TABLE table=test-table [id=1a1ec75518914948a641d0a50e094b3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:25:12.228060 20767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8701d169015d41b28db0413d7ebef952. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:12.228147 20499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8701d169015d41b28db0413d7ebef952. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:12.228320 20633 tablet_service.cc:1467] Processing CreateTablet for tablet 8701d169015d41b28db0413d7ebef952 (DEFAULT_TABLE table=test-table [id=1a1ec75518914948a641d0a50e094b3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:25:12.228629 20633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8701d169015d41b28db0413d7ebef952. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:12.231137 20850 tablet_bootstrap.cc:492] T 8701d169015d41b28db0413d7ebef952 P 84446b39131540478e4d28ca20bccaf3: Bootstrap starting.
I20260928 12:25:12.231343 20852 tablet_bootstrap.cc:492] T 8701d169015d41b28db0413d7ebef952 P 385640e696a74e0f91a73776a38fff68: Bootstrap starting.
I20260928 12:25:12.231514 20851 tablet_bootstrap.cc:492] T 8701d169015d41b28db0413d7ebef952 P aaf6099376104ccdaddefc70508adeca: Bootstrap starting.
I20260928 12:25:12.231789 20850 tablet_bootstrap.cc:654] T 8701d169015d41b28db0413d7ebef952 P 84446b39131540478e4d28ca20bccaf3: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:12.231985 20852 tablet_bootstrap.cc:654] T 8701d169015d41b28db0413d7ebef952 P 385640e696a74e0f91a73776a38fff68: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:12.232048 20851 tablet_bootstrap.cc:654] T 8701d169015d41b28db0413d7ebef952 P aaf6099376104ccdaddefc70508adeca: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:12.232273 20850 log.cc:824] T 8701d169015d41b28db0413d7ebef952 P 84446b39131540478e4d28ca20bccaf3: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:12.232537 20851 log.cc:824] T 8701d169015d41b28db0413d7ebef952 P aaf6099376104ccdaddefc70508adeca: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:12.232537 20852 log.cc:824] T 8701d169015d41b28db0413d7ebef952 P 385640e696a74e0f91a73776a38fff68: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:12.232972 20850 tablet_bootstrap.cc:492] T 8701d169015d41b28db0413d7ebef952 P 84446b39131540478e4d28ca20bccaf3: No bootstrap required, opened a new log
I20260928 12:25:12.233045 20850 ts_tablet_manager.cc:1397] T 8701d169015d41b28db0413d7ebef952 P 84446b39131540478e4d28ca20bccaf3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:25:12.233207 20851 tablet_bootstrap.cc:492] T 8701d169015d41b28db0413d7ebef952 P aaf6099376104ccdaddefc70508adeca: No bootstrap required, opened a new log
I20260928 12:25:12.233306 20851 ts_tablet_manager.cc:1397] T 8701d169015d41b28db0413d7ebef952 P aaf6099376104ccdaddefc70508adeca: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:25:12.233577 20852 tablet_bootstrap.cc:492] T 8701d169015d41b28db0413d7ebef952 P 385640e696a74e0f91a73776a38fff68: No bootstrap required, opened a new log
I20260928 12:25:12.233675 20852 ts_tablet_manager.cc:1397] T 8701d169015d41b28db0413d7ebef952 P 385640e696a74e0f91a73776a38fff68: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:25:12.234536 20850 raft_consensus.cc:348] T 8701d169015d41b28db0413d7ebef952 P 84446b39131540478e4d28ca20bccaf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "385640e696a74e0f91a73776a38fff68" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 41277 } } peers { permanent_uuid: "84446b39131540478e4d28ca20bccaf3" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34759 } } peers { permanent_uuid: "aaf6099376104ccdaddefc70508adeca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45845 } }
I20260928 12:25:12.234679 20850 raft_consensus.cc:374] T 8701d169015d41b28db0413d7ebef952 P 84446b39131540478e4d28ca20bccaf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:12.234719 20850 raft_consensus.cc:729] T 8701d169015d41b28db0413d7ebef952 P 84446b39131540478e4d28ca20bccaf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84446b39131540478e4d28ca20bccaf3, State: Initialized, Role: FOLLOWER
I20260928 12:25:12.234840 20850 consensus_queue.cc:260] T 8701d169015d41b28db0413d7ebef952 P 84446b39131540478e4d28ca20bccaf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "385640e696a74e0f91a73776a38fff68" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 41277 } } peers { permanent_uuid: "84446b39131540478e4d28ca20bccaf3" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34759 } } peers { permanent_uuid: "aaf6099376104ccdaddefc70508adeca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45845 } }
I20260928 12:25:12.235076 20850 ts_tablet_manager.cc:1428] T 8701d169015d41b28db0413d7ebef952 P 84446b39131540478e4d28ca20bccaf3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:25:12.235199 20699 heartbeater.cc:503] Master 127.2.56.62:33349 was elected leader, sending a full tablet report...
I20260928 12:25:12.235097 20851 raft_consensus.cc:348] T 8701d169015d41b28db0413d7ebef952 P aaf6099376104ccdaddefc70508adeca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "385640e696a74e0f91a73776a38fff68" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 41277 } } peers { permanent_uuid: "84446b39131540478e4d28ca20bccaf3" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34759 } } peers { permanent_uuid: "aaf6099376104ccdaddefc70508adeca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45845 } }
I20260928 12:25:12.235263 20851 raft_consensus.cc:374] T 8701d169015d41b28db0413d7ebef952 P aaf6099376104ccdaddefc70508adeca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:12.235309 20851 raft_consensus.cc:729] T 8701d169015d41b28db0413d7ebef952 P aaf6099376104ccdaddefc70508adeca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf6099376104ccdaddefc70508adeca, State: Initialized, Role: FOLLOWER
I20260928 12:25:12.235431 20851 consensus_queue.cc:260] T 8701d169015d41b28db0413d7ebef952 P aaf6099376104ccdaddefc70508adeca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "385640e696a74e0f91a73776a38fff68" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 41277 } } peers { permanent_uuid: "84446b39131540478e4d28ca20bccaf3" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34759 } } peers { permanent_uuid: "aaf6099376104ccdaddefc70508adeca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45845 } }
I20260928 12:25:12.235538 20852 raft_consensus.cc:348] T 8701d169015d41b28db0413d7ebef952 P 385640e696a74e0f91a73776a38fff68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "385640e696a74e0f91a73776a38fff68" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 41277 } } peers { permanent_uuid: "84446b39131540478e4d28ca20bccaf3" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34759 } } peers { permanent_uuid: "aaf6099376104ccdaddefc70508adeca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45845 } }
I20260928 12:25:12.235684 20852 raft_consensus.cc:374] T 8701d169015d41b28db0413d7ebef952 P 385640e696a74e0f91a73776a38fff68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:12.235715 20852 raft_consensus.cc:729] T 8701d169015d41b28db0413d7ebef952 P 385640e696a74e0f91a73776a38fff68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 385640e696a74e0f91a73776a38fff68, State: Initialized, Role: FOLLOWER
I20260928 12:25:12.235720 20833 heartbeater.cc:503] Master 127.2.56.62:33349 was elected leader, sending a full tablet report...
I20260928 12:25:12.235843 20852 consensus_queue.cc:260] T 8701d169015d41b28db0413d7ebef952 P 385640e696a74e0f91a73776a38fff68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "385640e696a74e0f91a73776a38fff68" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 41277 } } peers { permanent_uuid: "84446b39131540478e4d28ca20bccaf3" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34759 } } peers { permanent_uuid: "aaf6099376104ccdaddefc70508adeca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45845 } }
I20260928 12:25:12.235951 20851 ts_tablet_manager.cc:1428] T 8701d169015d41b28db0413d7ebef952 P aaf6099376104ccdaddefc70508adeca: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 12:25:12.236105 20852 ts_tablet_manager.cc:1428] T 8701d169015d41b28db0413d7ebef952 P 385640e696a74e0f91a73776a38fff68: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:25:12.236222 20565 heartbeater.cc:503] Master 127.2.56.62:33349 was elected leader, sending a full tablet report...
I20260928 12:25:12.243593 20856 raft_consensus.cc:482] T 8701d169015d41b28db0413d7ebef952 P 84446b39131540478e4d28ca20bccaf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:25:12.243660 20856 raft_consensus.cc:504] T 8701d169015d41b28db0413d7ebef952 P 84446b39131540478e4d28ca20bccaf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "385640e696a74e0f91a73776a38fff68" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 41277 } } peers { permanent_uuid: "84446b39131540478e4d28ca20bccaf3" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34759 } } peers { permanent_uuid: "aaf6099376104ccdaddefc70508adeca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45845 } }
I20260928 12:25:12.243909 20856 leader_election.cc:288] T 8701d169015d41b28db0413d7ebef952 P 84446b39131540478e4d28ca20bccaf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 385640e696a74e0f91a73776a38fff68 (127.2.56.1:41277), aaf6099376104ccdaddefc70508adeca (127.2.56.3:45845)
I20260928 12:25:12.247273 20849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:12.244084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:49571 (local address 127.2.56.3:45845)
0928 12:25:12.244132 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:12.244134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:12.244143 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:25:12.244198 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:12.244199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:12.244209 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:12.244259 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:25:12.244388 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.245127 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.246533 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.246966 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.247021 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:12.247031 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:12.247188 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 12:25:12.247218 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 12:25:12.247283 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:12.244187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:38899 (local address 127.2.56.1:41277)
0928 12:25:12.244234 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:12.244235 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:12.244269 (+ 34us) server_negotiation.cc:407] Connection header received
0928 12:25:12.244290 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:12.244291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:12.244299 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:12.244342 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:25:12.244451 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.245131 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.246545 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.246973 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.247038 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:12.247047 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:12.247177 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 12:25:12.247211 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 12:25:12.247287 20701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:12.244039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:45845 (local address 127.2.56.2:49571)
0928 12:25:12.244096 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:12.244108 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:12.244145 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:12.244266 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:12.244267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:12.244268 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:25:12.244328 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:12.244330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.245209 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.245210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:12.246457 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:12.246458 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.247039 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.247040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:12.247119 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:12.247122 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:25:12.247169 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 12:25:12.247214 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":14}
I20260928 12:25:12.247289 20859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:12.244039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:41277 (local address 127.2.56.2:38899)
0928 12:25:12.244185 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:12.244195 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:12.244243 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:12.244343 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:12.244344 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:12.244345 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:25:12.244397 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:12.244399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.245203 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.245203 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:12.246435 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:12.246439 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.247051 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.247052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:12.247117 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:12.247121 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:25:12.247165 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 12:25:12.247205 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":72,"thread_start_us":79,"threads_started":1}
I20260928 12:25:12.247651 20787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8701d169015d41b28db0413d7ebef952" candidate_uuid: "84446b39131540478e4d28ca20bccaf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf6099376104ccdaddefc70508adeca" is_pre_election: true
I20260928 12:25:12.247778 20787 raft_consensus.cc:2454] T 8701d169015d41b28db0413d7ebef952 P aaf6099376104ccdaddefc70508adeca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84446b39131540478e4d28ca20bccaf3 in term 0.
I20260928 12:25:12.247788 20519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8701d169015d41b28db0413d7ebef952" candidate_uuid: "84446b39131540478e4d28ca20bccaf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "385640e696a74e0f91a73776a38fff68" is_pre_election: true
I20260928 12:25:12.247901 20519 raft_consensus.cc:2454] T 8701d169015d41b28db0413d7ebef952 P 385640e696a74e0f91a73776a38fff68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84446b39131540478e4d28ca20bccaf3 in term 0.
I20260928 12:25:12.248085 20589 leader_election.cc:302] T 8701d169015d41b28db0413d7ebef952 P 84446b39131540478e4d28ca20bccaf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 385640e696a74e0f91a73776a38fff68, 84446b39131540478e4d28ca20bccaf3; no voters:
I20260928 12:25:12.248204 20856 raft_consensus.cc:2781] T 8701d169015d41b28db0413d7ebef952 P 84446b39131540478e4d28ca20bccaf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:25:12.248268 20856 raft_consensus.cc:482] T 8701d169015d41b28db0413d7ebef952 P 84446b39131540478e4d28ca20bccaf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:25:12.248288 20856 raft_consensus.cc:3037] T 8701d169015d41b28db0413d7ebef952 P 84446b39131540478e4d28ca20bccaf3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:12.248893 20856 raft_consensus.cc:504] T 8701d169015d41b28db0413d7ebef952 P 84446b39131540478e4d28ca20bccaf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "385640e696a74e0f91a73776a38fff68" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 41277 } } peers { permanent_uuid: "84446b39131540478e4d28ca20bccaf3" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34759 } } peers { permanent_uuid: "aaf6099376104ccdaddefc70508adeca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45845 } }
I20260928 12:25:12.248996 20856 leader_election.cc:288] T 8701d169015d41b28db0413d7ebef952 P 84446b39131540478e4d28ca20bccaf3 [CANDIDATE]: Term 1 election: Requested vote from peers 385640e696a74e0f91a73776a38fff68 (127.2.56.1:41277), aaf6099376104ccdaddefc70508adeca (127.2.56.3:45845)
I20260928 12:25:12.249114 20519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8701d169015d41b28db0413d7ebef952" candidate_uuid: "84446b39131540478e4d28ca20bccaf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "385640e696a74e0f91a73776a38fff68"
I20260928 12:25:12.249183 20519 raft_consensus.cc:3037] T 8701d169015d41b28db0413d7ebef952 P 385640e696a74e0f91a73776a38fff68 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:12.249903 20519 raft_consensus.cc:2454] T 8701d169015d41b28db0413d7ebef952 P 385640e696a74e0f91a73776a38fff68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84446b39131540478e4d28ca20bccaf3 in term 1.
I20260928 12:25:12.250007 20787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8701d169015d41b28db0413d7ebef952" candidate_uuid: "84446b39131540478e4d28ca20bccaf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf6099376104ccdaddefc70508adeca"
I20260928 12:25:12.250056 20589 leader_election.cc:302] T 8701d169015d41b28db0413d7ebef952 P 84446b39131540478e4d28ca20bccaf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 385640e696a74e0f91a73776a38fff68, 84446b39131540478e4d28ca20bccaf3; no voters:
I20260928 12:25:12.250077 20787 raft_consensus.cc:3037] T 8701d169015d41b28db0413d7ebef952 P aaf6099376104ccdaddefc70508adeca [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:12.250147 20856 raft_consensus.cc:2781] T 8701d169015d41b28db0413d7ebef952 P 84446b39131540478e4d28ca20bccaf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:25:12.250305 20856 raft_consensus.cc:686] T 8701d169015d41b28db0413d7ebef952 P 84446b39131540478e4d28ca20bccaf3 [term 1 LEADER]: Becoming Leader. State: Replica: 84446b39131540478e4d28ca20bccaf3, State: Running, Role: LEADER
I20260928 12:25:12.250380 20856 consensus_queue.cc:237] T 8701d169015d41b28db0413d7ebef952 P 84446b39131540478e4d28ca20bccaf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "385640e696a74e0f91a73776a38fff68" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 41277 } } peers { permanent_uuid: "84446b39131540478e4d28ca20bccaf3" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34759 } } peers { permanent_uuid: "aaf6099376104ccdaddefc70508adeca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45845 } }
I20260928 12:25:12.250833 20787 raft_consensus.cc:2454] T 8701d169015d41b28db0413d7ebef952 P aaf6099376104ccdaddefc70508adeca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84446b39131540478e4d28ca20bccaf3 in term 1.
I20260928 12:25:12.251088 20373 catalog_manager.cc:5495] T 8701d169015d41b28db0413d7ebef952 P 84446b39131540478e4d28ca20bccaf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84446b39131540478e4d28ca20bccaf3 (127.2.56.2). New cstate: current_term: 1 leader_uuid: "84446b39131540478e4d28ca20bccaf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "385640e696a74e0f91a73776a38fff68" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 41277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84446b39131540478e4d28ca20bccaf3" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaf6099376104ccdaddefc70508adeca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45845 } health_report { overall_health: UNKNOWN } } }
I20260928 12:25:12.256531 20848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:12.254580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55498 (local address 127.2.56.2:34759)
0928 12:25:12.254619 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:12.254620 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:12.254627 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:25:12.254676 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:12.254677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:12.254691 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:12.254742 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:25:12.254826 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.255270 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.255966 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.256138 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.256179 (+ 41us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:12.256286 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:25:12.256363 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:25:12.256425 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:12.256457 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 12:25:12.256464 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 12:25:12.258064 20787 raft_consensus.cc:1261] T 8701d169015d41b28db0413d7ebef952 P aaf6099376104ccdaddefc70508adeca [term 1 FOLLOWER]: Refusing update from remote peer 84446b39131540478e4d28ca20bccaf3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:25:12.258100 20519 raft_consensus.cc:1261] T 8701d169015d41b28db0413d7ebef952 P 385640e696a74e0f91a73776a38fff68 [term 1 FOLLOWER]: Refusing update from remote peer 84446b39131540478e4d28ca20bccaf3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:25:12.258387 20856 consensus_queue.cc:1035] T 8701d169015d41b28db0413d7ebef952 P 84446b39131540478e4d28ca20bccaf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "385640e696a74e0f91a73776a38fff68" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 41277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:25:12.258525 20860 consensus_queue.cc:1035] T 8701d169015d41b28db0413d7ebef952 P 84446b39131540478e4d28ca20bccaf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaf6099376104ccdaddefc70508adeca" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 45845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:25:12.260979 20865 mvcc.cc:205] Tried to move back new op lower bound from 7334290687006900224 to 7334290686978097152. Current Snapshot: MvccSnapshot[applied={T|T < 7334290687006900224}]
I20260928 12:25:12.261607 20864 mvcc.cc:205] Tried to move back new op lower bound from 7334290687006900224 to 7334290686978097152. Current Snapshot: MvccSnapshot[applied={T|T < 7334290687006900224 or (T in {7334290687006900224})}]
I20260928 12:25:12.262665 20374 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:25:12.262764 20374 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:25:12.263514 20374 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ca906222185497aab5ce1cfea75a2cd: Sending DeleteTablet for 3 replicas of tablet 8701d169015d41b28db0413d7ebef952
I20260928 12:25:12.263808 20633 tablet_service.cc:1514] Processing DeleteTablet for tablet 8701d169015d41b28db0413d7ebef952 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:25:12 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:55482
I20260928 12:25:12.263913 20767 tablet_service.cc:1514] Processing DeleteTablet for tablet 8701d169015d41b28db0413d7ebef952 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:25:12 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:38864
I20260928 12:25:12.263979 20499 tablet_service.cc:1514] Processing DeleteTablet for tablet 8701d169015d41b28db0413d7ebef952 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:25:12 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:52518
I20260928 12:25:12.264145 20873 tablet_replica.cc:330] T 8701d169015d41b28db0413d7ebef952 P aaf6099376104ccdaddefc70508adeca: stopping tablet replica
I20260928 12:25:12.264256 20873 raft_consensus.cc:2229] T 8701d169015d41b28db0413d7ebef952 P aaf6099376104ccdaddefc70508adeca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:25:12.264349 20873 raft_consensus.cc:2258] T 8701d169015d41b28db0413d7ebef952 P aaf6099376104ccdaddefc70508adeca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:25:12.264400 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 20436
I20260928 12:25:12.264762 20873 ts_tablet_manager.cc:1905] T 8701d169015d41b28db0413d7ebef952 P aaf6099376104ccdaddefc70508adeca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:25:12.266467 20873 ts_tablet_manager.cc:1918] T 8701d169015d41b28db0413d7ebef952 P aaf6099376104ccdaddefc70508adeca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:25:12.266536 20873 log.cc:1196] T 8701d169015d41b28db0413d7ebef952 P aaf6099376104ccdaddefc70508adeca: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-2/wal/wals/8701d169015d41b28db0413d7ebef952
I20260928 12:25:12.267030 20873 ts_tablet_manager.cc:1939] T 8701d169015d41b28db0413d7ebef952 P aaf6099376104ccdaddefc70508adeca: Deleting consensus metadata
I20260928 12:25:12.267088 20872 tablet_replica.cc:330] T 8701d169015d41b28db0413d7ebef952 P 84446b39131540478e4d28ca20bccaf3: stopping tablet replica
I20260928 12:25:12.267203 20872 raft_consensus.cc:2229] T 8701d169015d41b28db0413d7ebef952 P 84446b39131540478e4d28ca20bccaf3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:25:12.267297 20872 raft_consensus.cc:2258] T 8701d169015d41b28db0413d7ebef952 P 84446b39131540478e4d28ca20bccaf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:25:12.267469 20358 catalog_manager.cc:4841] TS aaf6099376104ccdaddefc70508adeca (127.2.56.3:45845): tablet 8701d169015d41b28db0413d7ebef952 (table test-table [id=1a1ec75518914948a641d0a50e094b3f]) successfully deleted
I20260928 12:25:12.267907 20872 ts_tablet_manager.cc:1905] T 8701d169015d41b28db0413d7ebef952 P 84446b39131540478e4d28ca20bccaf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:25:12.269990 20872 ts_tablet_manager.cc:1918] T 8701d169015d41b28db0413d7ebef952 P 84446b39131540478e4d28ca20bccaf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:25:12.270066 20872 log.cc:1196] T 8701d169015d41b28db0413d7ebef952 P 84446b39131540478e4d28ca20bccaf3: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598072638673-2272-0/minicluster-data/ts-1/wal/wals/8701d169015d41b28db0413d7ebef952
W20260928 12:25:12.270222 20361 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.56.1:41277 (error 108)
I20260928 12:25:12.270310 20872 ts_tablet_manager.cc:1939] T 8701d169015d41b28db0413d7ebef952 P 84446b39131540478e4d28ca20bccaf3: Deleting consensus metadata
I20260928 12:25:12.270639 20844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:12.270541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:41277 (local address 127.0.0.1:52528)
0928 12:25:12.270582 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:12.270609 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.1:41277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 12:25:12.270673 20359 catalog_manager.cc:4841] TS 84446b39131540478e4d28ca20bccaf3 (127.2.56.2:34759): tablet 8701d169015d41b28db0413d7ebef952 (table test-table [id=1a1ec75518914948a641d0a50e094b3f]) successfully deleted
I20260928 12:25:12.270731 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 20570
W20260928 12:25:12.270848 20361 catalog_manager.cc:4584] TS 385640e696a74e0f91a73776a38fff68 (127.2.56.1:41277): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8701d169015d41b28db0413d7ebef952: Network error: Client connection negotiation failed: client connection to 127.2.56.1:41277: connect: Connection refused (error 111)
I20260928 12:25:12.275331 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 20704
I20260928 12:25:12.279731 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 20342
2026-09-28T12:25:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1830 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T12:25:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:25:12Z Disabled control of system clock
I20260928 12:25:12.304419 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:46271
--webserver_interface=127.2.56.62
--webserver_port=0
--builtin_ntp_servers=127.2.56.20:34987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:46271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 12:25:12.383287 20879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:12.383517 20879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:12.383538 20879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:12.384663 20879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:25:12.384696 20879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:12.384708 20879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:12.384718 20879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:25:12.384730 20879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:25:12.386075 20879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:34987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:46271
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:46271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:12.386286 20879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:12.386524 20879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:12.391093 20887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:12.391136 20885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:12.391204 20884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:12.391338 20879 server_base.cc:1034] running on GCE node
I20260928 12:25:12.391566 20879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:12.391862 20879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:12.392997 20879 hybrid_clock.cc:648] HybridClock initialized: now 1790598312392983 us; error 35 us; skew 500 ppm
I20260928 12:25:12.394140 20879 webserver.cc:460] Webserver started at http://127.2.56.62:34017/ using document root <none> and password file <none>
I20260928 12:25:12.394351 20879 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:12.394393 20879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:12.394506 20879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:12.395548 20879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "1586b2bb30ef4d92b7436770f4c4ad87"
format_stamp: "Formatted at 2026-09-28 12:25:12 on dist-test-slave-920p"
server_key: "313def353130c384d55fd0ce1f7cff66"
server_key_iv: "26a94750cf973e34f75e3f8a74309097"
server_key_version: "encryptionkey@0"
I20260928 12:25:12.395885 20879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "1586b2bb30ef4d92b7436770f4c4ad87"
format_stamp: "Formatted at 2026-09-28 12:25:12 on dist-test-slave-920p"
server_key: "313def353130c384d55fd0ce1f7cff66"
server_key_iv: "26a94750cf973e34f75e3f8a74309097"
server_key_version: "encryptionkey@0"
I20260928 12:25:12.397184 20879 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:25:12.397996 20893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:12.398152 20879 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:25:12.398236 20879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "1586b2bb30ef4d92b7436770f4c4ad87"
format_stamp: "Formatted at 2026-09-28 12:25:12 on dist-test-slave-920p"
server_key: "313def353130c384d55fd0ce1f7cff66"
server_key_iv: "26a94750cf973e34f75e3f8a74309097"
server_key_version: "encryptionkey@0"
I20260928 12:25:12.398308 20879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:12.422497 20879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:12.422829 20879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:12.422960 20879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:12.427511 20879 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:46271
I20260928 12:25:12.427716 20945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:46271 every 8 connection(s)
I20260928 12:25:12.427930 20879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:25:12.428496 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 20879
I20260928 12:25:12.428598 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:25:12.428813 2272 external_mini_cluster.cc:1428] Setting key 1b17c51f1b1ae9aeff75fae43556d54c
I20260928 12:25:12.429150 20946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:12.432670 20946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1586b2bb30ef4d92b7436770f4c4ad87: Bootstrap starting.
I20260928 12:25:12.433310 20946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1586b2bb30ef4d92b7436770f4c4ad87: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:12.433615 20946 log.cc:824] T 00000000000000000000000000000000 P 1586b2bb30ef4d92b7436770f4c4ad87: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:12.434482 20946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1586b2bb30ef4d92b7436770f4c4ad87: No bootstrap required, opened a new log
I20260928 12:25:12.435336 20948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:12.429458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49010 (local address 127.2.56.62:46271)
0928 12:25:12.429648 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:12.429656 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:12.429770 (+ 114us) server_negotiation.cc:407] Connection header received
0928 12:25:12.430450 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:12.430461 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:12.430544 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:12.430705 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:12.430931 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.432237 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.433110 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.433718 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.433845 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:12.434590 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:12.434594 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:12.434595 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:12.434802 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:12.434843 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:12.434857 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:12.434924 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:12.435034 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 12:25:12.435119 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20260928 12:25:12.436533 20946 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1586b2bb30ef4d92b7436770f4c4ad87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1586b2bb30ef4d92b7436770f4c4ad87" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 46271 } }
I20260928 12:25:12.436671 20946 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1586b2bb30ef4d92b7436770f4c4ad87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:12.436693 20946 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1586b2bb30ef4d92b7436770f4c4ad87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1586b2bb30ef4d92b7436770f4c4ad87, State: Initialized, Role: FOLLOWER
I20260928 12:25:12.436793 20946 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1586b2bb30ef4d92b7436770f4c4ad87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1586b2bb30ef4d92b7436770f4c4ad87" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 46271 } }
I20260928 12:25:12.436864 20946 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1586b2bb30ef4d92b7436770f4c4ad87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:25:12.436892 20946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1586b2bb30ef4d92b7436770f4c4ad87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:25:12.436920 20946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1586b2bb30ef4d92b7436770f4c4ad87 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:12.437556 20946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1586b2bb30ef4d92b7436770f4c4ad87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1586b2bb30ef4d92b7436770f4c4ad87" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 46271 } }
I20260928 12:25:12.437651 20946 leader_election.cc:302] T 00000000000000000000000000000000 P 1586b2bb30ef4d92b7436770f4c4ad87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1586b2bb30ef4d92b7436770f4c4ad87; no voters:
I20260928 12:25:12.437832 20946 leader_election.cc:288] T 00000000000000000000000000000000 P 1586b2bb30ef4d92b7436770f4c4ad87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:25:12.437885 20951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1586b2bb30ef4d92b7436770f4c4ad87 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:25:12.438041 20951 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1586b2bb30ef4d92b7436770f4c4ad87 [term 1 LEADER]: Becoming Leader. State: Replica: 1586b2bb30ef4d92b7436770f4c4ad87, State: Running, Role: LEADER
I20260928 12:25:12.438069 20946 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1586b2bb30ef4d92b7436770f4c4ad87 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:25:12.438125 20951 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1586b2bb30ef4d92b7436770f4c4ad87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1586b2bb30ef4d92b7436770f4c4ad87" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 46271 } }
I20260928 12:25:12.438459 20952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1586b2bb30ef4d92b7436770f4c4ad87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1586b2bb30ef4d92b7436770f4c4ad87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1586b2bb30ef4d92b7436770f4c4ad87" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 46271 } } }
I20260928 12:25:12.438483 20953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1586b2bb30ef4d92b7436770f4c4ad87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1586b2bb30ef4d92b7436770f4c4ad87. Latest consensus state: current_term: 1 leader_uuid: "1586b2bb30ef4d92b7436770f4c4ad87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1586b2bb30ef4d92b7436770f4c4ad87" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 46271 } } }
I20260928 12:25:12.438545 20952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1586b2bb30ef4d92b7436770f4c4ad87 [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:12.438549 20953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1586b2bb30ef4d92b7436770f4c4ad87 [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:12.438742 20958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:25:12.439205 20958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:25:12.440521 20958 catalog_manager.cc:1347] Generated new cluster ID: 88670a4e250d496ebde04942a07b6cfb
I20260928 12:25:12.440572 20958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:25:12.452378 20958 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:25:12.452847 20958 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:25:12.456264 20958 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1586b2bb30ef4d92b7436770f4c4ad87: Generated new TSK 0
I20260928 12:25:12.456424 20958 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:25:12.464596 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.1:0
--local_ip_for_outbound_sockets=127.2.56.1
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:46271
--builtin_ntp_servers=127.2.56.20:34987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 12:25:12.544878 20970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:12.545081 20970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:12.545102 20970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:12.546291 20970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:12.546329 20970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:12.546355 20970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.1
I20260928 12:25:12.547641 20970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:34987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:46271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.1
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:12.547852 20970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:12.548125 20970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:12.553702 20976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:12.553790 20970 server_base.cc:1034] running on GCE node
W20260928 12:25:12.553900 20975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:12.553701 20978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:12.554096 20970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:12.554324 20970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:12.555470 20970 hybrid_clock.cc:648] HybridClock initialized: now 1790598312555461 us; error 25 us; skew 500 ppm
I20260928 12:25:12.556567 20970 webserver.cc:460] Webserver started at http://127.2.56.1:44747/ using document root <none> and password file <none>
I20260928 12:25:12.556771 20970 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:12.556818 20970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:12.556941 20970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:12.557996 20970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-0/data/instance:
uuid: "9368997abd4647a194dbef0de676224f"
format_stamp: "Formatted at 2026-09-28 12:25:12 on dist-test-slave-920p"
server_key: "81d324cadc5e7cddcf568d34333e10e8"
server_key_iv: "5656ec6606f25a55013ac6c9f197c905"
server_key_version: "encryptionkey@0"
I20260928 12:25:12.558308 20970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance:
uuid: "9368997abd4647a194dbef0de676224f"
format_stamp: "Formatted at 2026-09-28 12:25:12 on dist-test-slave-920p"
server_key: "81d324cadc5e7cddcf568d34333e10e8"
server_key_iv: "5656ec6606f25a55013ac6c9f197c905"
server_key_version: "encryptionkey@0"
I20260928 12:25:12.559547 20970 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 12:25:12.560194 20984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:12.560345 20970 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:25:12.560439 20970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-0/wal
uuid: "9368997abd4647a194dbef0de676224f"
format_stamp: "Formatted at 2026-09-28 12:25:12 on dist-test-slave-920p"
server_key: "81d324cadc5e7cddcf568d34333e10e8"
server_key_iv: "5656ec6606f25a55013ac6c9f197c905"
server_key_version: "encryptionkey@0"
I20260928 12:25:12.560508 20970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:12.570858 20970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:12.571106 20970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:12.571245 20970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:12.571480 20970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:25:12.571846 20970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:25:12.571892 20970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:12.571939 20970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:25:12.571967 20970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:12.577725 20970 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.1:37749
I20260928 12:25:12.577785 21097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.1:37749 every 8 connection(s)
I20260928 12:25:12.578073 20970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
I20260928 12:25:12.579278 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 20970
I20260928 12:25:12.579362 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance
I20260928 12:25:12.579532 2272 external_mini_cluster.cc:1428] Setting key abf90ee0f67456f7e57ca71e19143ac2
I20260928 12:25:12.580408 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.2:0
--local_ip_for_outbound_sockets=127.2.56.2
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:46271
--builtin_ntp_servers=127.2.56.20:34987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 12:25:12.583141 20948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:12.578741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:42055 (local address 127.2.56.62:46271)
0928 12:25:12.578786 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:12.578788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:12.579000 (+ 212us) server_negotiation.cc:407] Connection header received
0928 12:25:12.579538 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:12.579540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:12.579555 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:12.579622 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:12.580331 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.580863 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.581857 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.582053 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.582107 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:12.582879 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:12.582880 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:12.582881 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:12.582885 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:12.582899 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:12.582903 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:12.582947 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:12.583055 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 12:25:12.583082 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:25:12.583339 21100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:12.578723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:46271 (local address 127.2.56.1:42055)
0928 12:25:12.578896 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:12.578926 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:12.579450 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:12.579675 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:12.579679 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:12.579688 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:12.580250 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:12.580256 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.580945 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.580946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:12.581783 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:12.581786 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.582119 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.582120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:12.582247 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:12.582731 (+ 484us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:25:12.582732 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:12.582769 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:25:12.582795 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:12.582956 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:12.582961 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:12.582963 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:25:12.583024 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 12:25:12.583125 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
I20260928 12:25:12.583887 21098 heartbeater.cc:347] Connected to a master server at 127.2.56.62:46271
I20260928 12:25:12.583971 21098 heartbeater.cc:464] Registering TS with master...
I20260928 12:25:12.584142 21098 heartbeater.cc:511] Master 127.2.56.62:46271 requested a full tablet report, sending...
I20260928 12:25:12.584678 20910 ts_manager.cc:194] Registered new tserver with Master: 9368997abd4647a194dbef0de676224f (127.2.56.1:37749)
I20260928 12:25:12.585588 20910 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.56.1:42055
W20260928 12:25:12.660512 21101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:12.660718 21101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:12.660738 21101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:12.661916 21101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:12.661971 21101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:12.661998 21101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.2
I20260928 12:25:12.663371 21101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:34987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:46271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.2
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:12.663578 21101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:12.663841 21101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:12.668826 21106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:12.668790 21109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:12.668790 21107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:12.668861 21101 server_base.cc:1034] running on GCE node
I20260928 12:25:12.669193 21101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:12.669440 21101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:12.670593 21101 hybrid_clock.cc:648] HybridClock initialized: now 1790598312670581 us; error 27 us; skew 500 ppm
I20260928 12:25:12.671782 21101 webserver.cc:460] Webserver started at http://127.2.56.2:40969/ using document root <none> and password file <none>
I20260928 12:25:12.671991 21101 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:12.672034 21101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:12.672150 21101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:12.673187 21101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-1/data/instance:
uuid: "77983fbce2044cc897472ece4586b430"
format_stamp: "Formatted at 2026-09-28 12:25:12 on dist-test-slave-920p"
server_key: "104b1d2d1ad0dc12a3712f13a332b906"
server_key_iv: "97c10b265e983c79ea47432fe8bf4a7a"
server_key_version: "encryptionkey@0"
I20260928 12:25:12.673554 21101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance:
uuid: "77983fbce2044cc897472ece4586b430"
format_stamp: "Formatted at 2026-09-28 12:25:12 on dist-test-slave-920p"
server_key: "104b1d2d1ad0dc12a3712f13a332b906"
server_key_iv: "97c10b265e983c79ea47432fe8bf4a7a"
server_key_version: "encryptionkey@0"
I20260928 12:25:12.674814 21101 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:25:12.675544 21115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:12.675695 21101 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:25:12.675776 21101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-1/wal
uuid: "77983fbce2044cc897472ece4586b430"
format_stamp: "Formatted at 2026-09-28 12:25:12 on dist-test-slave-920p"
server_key: "104b1d2d1ad0dc12a3712f13a332b906"
server_key_iv: "97c10b265e983c79ea47432fe8bf4a7a"
server_key_version: "encryptionkey@0"
I20260928 12:25:12.675828 21101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:12.692929 21101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:12.693207 21101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:12.693352 21101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:12.693548 21101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:25:12.693833 21101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:25:12.693862 21101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:12.693888 21101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:25:12.693903 21101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:12.699086 21101 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.2:34687
I20260928 12:25:12.699162 21228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.2:34687 every 8 connection(s)
I20260928 12:25:12.699431 21101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
I20260928 12:25:12.704071 20948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:12.700077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:51665 (local address 127.2.56.62:46271)
0928 12:25:12.700157 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:12.700159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:12.700333 (+ 174us) server_negotiation.cc:407] Connection header received
0928 12:25:12.700820 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:12.700821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:12.700831 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:12.700870 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:12.701580 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.702062 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.702979 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.703166 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.703211 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:12.703880 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:12.703881 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:12.703881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:12.703885 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:12.703899 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:12.703902 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:12.703939 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:12.703992 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 12:25:12.704013 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 12:25:12.704249 21231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:12.700054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:46271 (local address 127.2.56.2:51665)
0928 12:25:12.700228 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:12.700262 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:12.700734 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:12.700932 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:12.700935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:12.700943 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:12.701510 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:12.701515 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.702142 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.702143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:12.702914 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:12.702916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.703227 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.703227 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:12.703327 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:12.703764 (+ 437us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:25:12.703764 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:12.703790 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:25:12.703812 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:12.703946 (+ 134us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:12.703948 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:12.703950 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:25:12.703994 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 12:25:12.704064 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":84,"threads_started":1}
I20260928 12:25:12.704644 21229 heartbeater.cc:347] Connected to a master server at 127.2.56.62:46271
I20260928 12:25:12.704730 21229 heartbeater.cc:464] Registering TS with master...
I20260928 12:25:12.704890 21229 heartbeater.cc:511] Master 127.2.56.62:46271 requested a full tablet report, sending...
I20260928 12:25:12.705209 20910 ts_manager.cc:194] Registered new tserver with Master: 77983fbce2044cc897472ece4586b430 (127.2.56.2:34687)
I20260928 12:25:12.705566 20910 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.56.2:51665
I20260928 12:25:12.705655 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 21101
I20260928 12:25:12.705746 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance
I20260928 12:25:12.705955 2272 external_mini_cluster.cc:1428] Setting key 3a61370730faf638895b05398918932c
I20260928 12:25:12.706796 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.3:0
--local_ip_for_outbound_sockets=127.2.56.3
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:46271
--builtin_ntp_servers=127.2.56.20:34987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 12:25:12.786581 21232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:12.786775 21232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:12.786796 21232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:12.787883 21232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:12.787911 21232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:12.787933 21232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.3
I20260928 12:25:12.789249 21232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:34987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:46271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.3
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:12.789464 21232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:12.789692 21232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:12.794234 21238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:12.794328 21240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:12.794333 21232 server_base.cc:1034] running on GCE node
W20260928 12:25:12.794415 21237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:12.794744 21232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:12.794967 21232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:12.796105 21232 hybrid_clock.cc:648] HybridClock initialized: now 1790598312796091 us; error 25 us; skew 500 ppm
I20260928 12:25:12.797224 21232 webserver.cc:460] Webserver started at http://127.2.56.3:32981/ using document root <none> and password file <none>
I20260928 12:25:12.797451 21232 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:12.797498 21232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:12.797631 21232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:12.798645 21232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-2/data/instance:
uuid: "bb9109131fb249d99f3ec728b912f961"
format_stamp: "Formatted at 2026-09-28 12:25:12 on dist-test-slave-920p"
server_key: "43ec2c8c66294c233c2a6d17433e541f"
server_key_iv: "ac6a0dbc9460f039f96d72b51cf14dfd"
server_key_version: "encryptionkey@0"
I20260928 12:25:12.798956 21232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance:
uuid: "bb9109131fb249d99f3ec728b912f961"
format_stamp: "Formatted at 2026-09-28 12:25:12 on dist-test-slave-920p"
server_key: "43ec2c8c66294c233c2a6d17433e541f"
server_key_iv: "ac6a0dbc9460f039f96d72b51cf14dfd"
server_key_version: "encryptionkey@0"
I20260928 12:25:12.800083 21232 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:25:12.800719 21246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:12.800860 21232 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:25:12.800947 21232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-2/wal
uuid: "bb9109131fb249d99f3ec728b912f961"
format_stamp: "Formatted at 2026-09-28 12:25:12 on dist-test-slave-920p"
server_key: "43ec2c8c66294c233c2a6d17433e541f"
server_key_iv: "ac6a0dbc9460f039f96d72b51cf14dfd"
server_key_version: "encryptionkey@0"
I20260928 12:25:12.801009 21232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:12.817605 21232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:12.817865 21232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:12.817982 21232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:12.818176 21232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:25:12.818457 21232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:25:12.818490 21232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:12.818516 21232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:25:12.818533 21232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:12.823537 21232 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.3:46137
I20260928 12:25:12.823589 21359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.3:46137 every 8 connection(s)
I20260928 12:25:12.823871 21232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
I20260928 12:25:12.828301 20948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:12.824470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:56271 (local address 127.2.56.62:46271)
0928 12:25:12.824530 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:12.824532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:12.824796 (+ 264us) server_negotiation.cc:407] Connection header received
0928 12:25:12.825232 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:12.825233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:12.825244 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:12.825351 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:12.825928 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.826415 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.827273 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.827442 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.827481 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:12.828114 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:12.828116 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:12.828116 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:12.828120 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:12.828133 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:12.828136 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:12.828173 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:12.828225 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 12:25:12.828245 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 12:25:12.828474 21362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:12.824456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:46271 (local address 127.2.56.3:56271)
0928 12:25:12.824622 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:12.824660 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:12.825125 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:12.825405 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:12.825409 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:12.825416 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:12.825866 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:12.825870 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.826475 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.826476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:12.827218 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:12.827220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.827492 (+ 272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.827493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:12.827595 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:12.828004 (+ 409us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:25:12.828004 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:12.828029 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:25:12.828048 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:12.828175 (+ 127us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:12.828177 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:12.828179 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:25:12.828219 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 12:25:12.828287 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
I20260928 12:25:12.828894 21360 heartbeater.cc:347] Connected to a master server at 127.2.56.62:46271
I20260928 12:25:12.828969 21360 heartbeater.cc:464] Registering TS with master...
I20260928 12:25:12.829128 21360 heartbeater.cc:511] Master 127.2.56.62:46271 requested a full tablet report, sending...
I20260928 12:25:12.829479 20910 ts_manager.cc:194] Registered new tserver with Master: bb9109131fb249d99f3ec728b912f961 (127.2.56.3:46137)
I20260928 12:25:12.829828 20910 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.56.3:56271
I20260928 12:25:12.831274 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 21232
I20260928 12:25:12.831346 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance
I20260928 12:25:12.831513 2272 external_mini_cluster.cc:1428] Setting key 69c606a64c0366091600473d69147e35
I20260928 12:25:12.832258 2272 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:25:12.836397 20948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:12.833496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49022 (local address 127.2.56.62:46271)
0928 12:25:12.833557 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:12.833559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:12.833639 (+ 80us) server_negotiation.cc:407] Connection header received
0928 12:25:12.833678 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:12.833678 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:12.833686 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:12.833725 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:12.833854 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.834296 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.835085 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.835296 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.835362 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:12.835936 (+ 574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:12.835937 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:12.835938 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:12.835942 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:12.835955 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:12.835959 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:12.836038 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:12.836288 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 12:25:12.836303 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 12:25:12.838308 20910 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:25:12.840067 20910 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:25:12.845877 21369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:12.841905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:46137 (local address 127.0.0.1:48170)
0928 12:25:12.842124 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:12.842162 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:12.842227 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:12.842518 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:12.842521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:12.842523 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:25:12.842590 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:12.842592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.843949 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.843950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:12.844790 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:12.844792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.845622 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.845623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:12.845682 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:12.845686 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:25:12.845724 (+ 38us) client_negotiation.cc:240] Negotiation successful
0928 12:25:12.845778 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":27,"thread_start_us":71,"threads_started":1}
I20260928 12:25:12.845876 21370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:12.841995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58518 (local address 127.2.56.1:37749)
0928 12:25:12.842371 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:12.842375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:12.842383 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:25:12.842416 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:12.842418 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:12.842431 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:12.842500 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:25:12.842722 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.843488 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.844856 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.845501 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.845562 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:12.845657 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:12.845763 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 12:25:12.845797 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":85,"threads_started":1}
I20260928 12:25:12.846103 21368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:12.841907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:37749 (local address 127.0.0.1:58518)
0928 12:25:12.842164 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:12.842171 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:12.842227 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:12.842587 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:12.842588 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:12.842590 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:25:12.842644 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:12.842645 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.843573 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.843574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:12.844761 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:12.844764 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.845585 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.845586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:12.845667 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:12.845680 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 12:25:12.845749 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 12:25:12.845795 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":11,"spinlock_wait_cycles":896,"thread_start_us":100,"threads_started":1}
I20260928 12:25:12.846212 21372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:12.842101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:34687 (local address 127.0.0.1:51642)
0928 12:25:12.842217 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:12.842227 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:12.842275 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:12.842613 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:12.842614 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:12.842616 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:25:12.842648 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:12.842650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.843938 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.843939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:12.845041 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:12.845044 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.845951 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.845952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:12.846034 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:12.846038 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:25:12.846066 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 12:25:12.846097 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":30,"thread_start_us":66,"threads_started":1}
I20260928 12:25:12.846382 21373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:12.842299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51642 (local address 127.2.56.2:34687)
0928 12:25:12.842453 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:12.842458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:12.842467 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:25:12.842505 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:12.842506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:12.842522 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:12.842718 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:25:12.842777 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.843846 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.845121 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.845868 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.846152 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:12.846240 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:12.846263 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 12:25:12.846301 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":82,"threads_started":1}
I20260928 12:25:12.846599 21371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:12.842012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48170 (local address 127.2.56.3:46137)
0928 12:25:12.842356 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:12.842361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:12.842371 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:25:12.842407 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:12.842409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:12.842425 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:12.842495 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:25:12.842698 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.843857 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.844871 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.845524 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.845841 (+ 317us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:12.846463 (+ 622us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:12.846488 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 12:25:12.846528 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":71,"threads_started":1}
I20260928 12:25:12.846971 21032 tablet_service.cc:1467] Processing CreateTablet for tablet 2a1b247f3b7f49409e568ef5e2d092a2 (DEFAULT_TABLE table=test-table [id=6bd684ae03384f47b72c71657f61e299]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:25:12.847002 21163 tablet_service.cc:1467] Processing CreateTablet for tablet 2a1b247f3b7f49409e568ef5e2d092a2 (DEFAULT_TABLE table=test-table [id=6bd684ae03384f47b72c71657f61e299]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:25:12.847102 21294 tablet_service.cc:1467] Processing CreateTablet for tablet 2a1b247f3b7f49409e568ef5e2d092a2 (DEFAULT_TABLE table=test-table [id=6bd684ae03384f47b72c71657f61e299]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:25:12.847321 21163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a1b247f3b7f49409e568ef5e2d092a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:12.847322 21032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a1b247f3b7f49409e568ef5e2d092a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:12.847378 21294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a1b247f3b7f49409e568ef5e2d092a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:12.849862 21374 tablet_bootstrap.cc:492] T 2a1b247f3b7f49409e568ef5e2d092a2 P 9368997abd4647a194dbef0de676224f: Bootstrap starting.
I20260928 12:25:12.850412 21375 tablet_bootstrap.cc:492] T 2a1b247f3b7f49409e568ef5e2d092a2 P 77983fbce2044cc897472ece4586b430: Bootstrap starting.
I20260928 12:25:12.850569 21374 tablet_bootstrap.cc:654] T 2a1b247f3b7f49409e568ef5e2d092a2 P 9368997abd4647a194dbef0de676224f: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:12.850931 21374 log.cc:824] T 2a1b247f3b7f49409e568ef5e2d092a2 P 9368997abd4647a194dbef0de676224f: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:12.851049 21375 tablet_bootstrap.cc:654] T 2a1b247f3b7f49409e568ef5e2d092a2 P 77983fbce2044cc897472ece4586b430: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:12.851308 21375 log.cc:824] T 2a1b247f3b7f49409e568ef5e2d092a2 P 77983fbce2044cc897472ece4586b430: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:12.851619 21376 tablet_bootstrap.cc:492] T 2a1b247f3b7f49409e568ef5e2d092a2 P bb9109131fb249d99f3ec728b912f961: Bootstrap starting.
I20260928 12:25:12.851660 21374 tablet_bootstrap.cc:492] T 2a1b247f3b7f49409e568ef5e2d092a2 P 9368997abd4647a194dbef0de676224f: No bootstrap required, opened a new log
I20260928 12:25:12.851711 21374 ts_tablet_manager.cc:1397] T 2a1b247f3b7f49409e568ef5e2d092a2 P 9368997abd4647a194dbef0de676224f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:25:12.851917 21375 tablet_bootstrap.cc:492] T 2a1b247f3b7f49409e568ef5e2d092a2 P 77983fbce2044cc897472ece4586b430: No bootstrap required, opened a new log
I20260928 12:25:12.851972 21375 ts_tablet_manager.cc:1397] T 2a1b247f3b7f49409e568ef5e2d092a2 P 77983fbce2044cc897472ece4586b430: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:25:12.852226 21376 tablet_bootstrap.cc:654] T 2a1b247f3b7f49409e568ef5e2d092a2 P bb9109131fb249d99f3ec728b912f961: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:12.852533 21376 log.cc:824] T 2a1b247f3b7f49409e568ef5e2d092a2 P bb9109131fb249d99f3ec728b912f961: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:12.853236 21376 tablet_bootstrap.cc:492] T 2a1b247f3b7f49409e568ef5e2d092a2 P bb9109131fb249d99f3ec728b912f961: No bootstrap required, opened a new log
I20260928 12:25:12.853322 21376 ts_tablet_manager.cc:1397] T 2a1b247f3b7f49409e568ef5e2d092a2 P bb9109131fb249d99f3ec728b912f961: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:25:12.854053 21375 raft_consensus.cc:348] T 2a1b247f3b7f49409e568ef5e2d092a2 P 77983fbce2044cc897472ece4586b430 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9109131fb249d99f3ec728b912f961" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 46137 } } peers { permanent_uuid: "9368997abd4647a194dbef0de676224f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37749 } } peers { permanent_uuid: "77983fbce2044cc897472ece4586b430" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34687 } }
I20260928 12:25:12.854178 21375 raft_consensus.cc:374] T 2a1b247f3b7f49409e568ef5e2d092a2 P 77983fbce2044cc897472ece4586b430 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:12.854211 21375 raft_consensus.cc:729] T 2a1b247f3b7f49409e568ef5e2d092a2 P 77983fbce2044cc897472ece4586b430 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77983fbce2044cc897472ece4586b430, State: Initialized, Role: FOLLOWER
I20260928 12:25:12.854106 21374 raft_consensus.cc:348] T 2a1b247f3b7f49409e568ef5e2d092a2 P 9368997abd4647a194dbef0de676224f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9109131fb249d99f3ec728b912f961" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 46137 } } peers { permanent_uuid: "9368997abd4647a194dbef0de676224f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37749 } } peers { permanent_uuid: "77983fbce2044cc897472ece4586b430" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34687 } }
I20260928 12:25:12.854257 21374 raft_consensus.cc:374] T 2a1b247f3b7f49409e568ef5e2d092a2 P 9368997abd4647a194dbef0de676224f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:12.854287 21374 raft_consensus.cc:729] T 2a1b247f3b7f49409e568ef5e2d092a2 P 9368997abd4647a194dbef0de676224f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9368997abd4647a194dbef0de676224f, State: Initialized, Role: FOLLOWER
I20260928 12:25:12.854338 21375 consensus_queue.cc:260] T 2a1b247f3b7f49409e568ef5e2d092a2 P 77983fbce2044cc897472ece4586b430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9109131fb249d99f3ec728b912f961" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 46137 } } peers { permanent_uuid: "9368997abd4647a194dbef0de676224f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37749 } } peers { permanent_uuid: "77983fbce2044cc897472ece4586b430" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34687 } }
I20260928 12:25:12.854398 21374 consensus_queue.cc:260] T 2a1b247f3b7f49409e568ef5e2d092a2 P 9368997abd4647a194dbef0de676224f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9109131fb249d99f3ec728b912f961" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 46137 } } peers { permanent_uuid: "9368997abd4647a194dbef0de676224f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37749 } } peers { permanent_uuid: "77983fbce2044cc897472ece4586b430" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34687 } }
I20260928 12:25:12.854557 21375 ts_tablet_manager.cc:1428] T 2a1b247f3b7f49409e568ef5e2d092a2 P 77983fbce2044cc897472ece4586b430: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:25:12.854641 21374 ts_tablet_manager.cc:1428] T 2a1b247f3b7f49409e568ef5e2d092a2 P 9368997abd4647a194dbef0de676224f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:25:12.854684 21229 heartbeater.cc:503] Master 127.2.56.62:46271 was elected leader, sending a full tablet report...
I20260928 12:25:12.854722 21098 heartbeater.cc:503] Master 127.2.56.62:46271 was elected leader, sending a full tablet report...
I20260928 12:25:12.856026 21376 raft_consensus.cc:348] T 2a1b247f3b7f49409e568ef5e2d092a2 P bb9109131fb249d99f3ec728b912f961 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9109131fb249d99f3ec728b912f961" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 46137 } } peers { permanent_uuid: "9368997abd4647a194dbef0de676224f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37749 } } peers { permanent_uuid: "77983fbce2044cc897472ece4586b430" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34687 } }
I20260928 12:25:12.856139 21376 raft_consensus.cc:374] T 2a1b247f3b7f49409e568ef5e2d092a2 P bb9109131fb249d99f3ec728b912f961 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:12.856158 21376 raft_consensus.cc:729] T 2a1b247f3b7f49409e568ef5e2d092a2 P bb9109131fb249d99f3ec728b912f961 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb9109131fb249d99f3ec728b912f961, State: Initialized, Role: FOLLOWER
I20260928 12:25:12.856256 21376 consensus_queue.cc:260] T 2a1b247f3b7f49409e568ef5e2d092a2 P bb9109131fb249d99f3ec728b912f961 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9109131fb249d99f3ec728b912f961" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 46137 } } peers { permanent_uuid: "9368997abd4647a194dbef0de676224f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37749 } } peers { permanent_uuid: "77983fbce2044cc897472ece4586b430" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34687 } }
I20260928 12:25:12.856459 21376 ts_tablet_manager.cc:1428] T 2a1b247f3b7f49409e568ef5e2d092a2 P bb9109131fb249d99f3ec728b912f961: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 12:25:12.856493 21360 heartbeater.cc:503] Master 127.2.56.62:46271 was elected leader, sending a full tablet report...
I20260928 12:25:12.859490 21382 raft_consensus.cc:482] T 2a1b247f3b7f49409e568ef5e2d092a2 P bb9109131fb249d99f3ec728b912f961 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:25:12.859562 21382 raft_consensus.cc:504] T 2a1b247f3b7f49409e568ef5e2d092a2 P bb9109131fb249d99f3ec728b912f961 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9109131fb249d99f3ec728b912f961" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 46137 } } peers { permanent_uuid: "9368997abd4647a194dbef0de676224f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37749 } } peers { permanent_uuid: "77983fbce2044cc897472ece4586b430" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34687 } }
I20260928 12:25:12.859818 21382 leader_election.cc:288] T 2a1b247f3b7f49409e568ef5e2d092a2 P bb9109131fb249d99f3ec728b912f961 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9368997abd4647a194dbef0de676224f (127.2.56.1:37749), 77983fbce2044cc897472ece4586b430 (127.2.56.2:34687)
I20260928 12:25:12.862684 21373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:12.860031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:37357 (local address 127.2.56.2:34687)
0928 12:25:12.860095 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:12.860097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:12.860230 (+ 133us) server_negotiation.cc:407] Connection header received
0928 12:25:12.860256 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:12.860256 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:12.860264 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:12.860304 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:25:12.860399 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.861075 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.862062 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.862396 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.862454 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:12.862462 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:12.862592 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 12:25:12.862622 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 12:25:12.862689 21370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:12.860024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:41813 (local address 127.2.56.1:37749)
0928 12:25:12.860056 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:12.860057 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:12.860065 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:25:12.860126 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:12.860127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:12.860137 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:12.860186 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:25:12.860329 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.861064 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.862020 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.862381 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.862433 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:12.862442 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:12.862581 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 12:25:12.862615 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 12:25:12.862701 21362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:12.859936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:37749 (local address 127.2.56.3:41813)
0928 12:25:12.860017 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:12.860028 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:12.860069 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:12.860210 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:12.860211 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:12.860213 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:25:12.860271 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:12.860272 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.861159 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.861160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:12.861943 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:12.861945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.862451 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.862452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:12.862518 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:12.862521 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:25:12.862575 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 12:25:12.862615 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":37}
I20260928 12:25:12.862712 21383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:12.859972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:34687 (local address 127.2.56.3:37357)
0928 12:25:12.860156 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:12.860177 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:12.860219 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:12.860305 (+ 86us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:12.860306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:12.860307 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:25:12.860354 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:12.860356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.861190 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.861191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:12.861975 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:12.861977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.862462 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.862463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:12.862532 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:12.862535 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:25:12.862580 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 12:25:12.862615 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":103,"threads_started":1}
I20260928 12:25:12.863041 21183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a1b247f3b7f49409e568ef5e2d092a2" candidate_uuid: "bb9109131fb249d99f3ec728b912f961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77983fbce2044cc897472ece4586b430" is_pre_election: true
I20260928 12:25:12.863170 21183 raft_consensus.cc:2454] T 2a1b247f3b7f49409e568ef5e2d092a2 P 77983fbce2044cc897472ece4586b430 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb9109131fb249d99f3ec728b912f961 in term 0.
I20260928 12:25:12.863238 21052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a1b247f3b7f49409e568ef5e2d092a2" candidate_uuid: "bb9109131fb249d99f3ec728b912f961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9368997abd4647a194dbef0de676224f" is_pre_election: true
I20260928 12:25:12.863366 21052 raft_consensus.cc:2454] T 2a1b247f3b7f49409e568ef5e2d092a2 P 9368997abd4647a194dbef0de676224f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb9109131fb249d99f3ec728b912f961 in term 0.
I20260928 12:25:12.863363 21248 leader_election.cc:302] T 2a1b247f3b7f49409e568ef5e2d092a2 P bb9109131fb249d99f3ec728b912f961 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77983fbce2044cc897472ece4586b430, bb9109131fb249d99f3ec728b912f961; no voters:
I20260928 12:25:12.863471 21382 raft_consensus.cc:2781] T 2a1b247f3b7f49409e568ef5e2d092a2 P bb9109131fb249d99f3ec728b912f961 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:25:12.863516 21382 raft_consensus.cc:482] T 2a1b247f3b7f49409e568ef5e2d092a2 P bb9109131fb249d99f3ec728b912f961 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:25:12.863538 21382 raft_consensus.cc:3037] T 2a1b247f3b7f49409e568ef5e2d092a2 P bb9109131fb249d99f3ec728b912f961 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:12.864631 21382 raft_consensus.cc:504] T 2a1b247f3b7f49409e568ef5e2d092a2 P bb9109131fb249d99f3ec728b912f961 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9109131fb249d99f3ec728b912f961" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 46137 } } peers { permanent_uuid: "9368997abd4647a194dbef0de676224f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37749 } } peers { permanent_uuid: "77983fbce2044cc897472ece4586b430" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34687 } }
I20260928 12:25:12.864769 21382 leader_election.cc:288] T 2a1b247f3b7f49409e568ef5e2d092a2 P bb9109131fb249d99f3ec728b912f961 [CANDIDATE]: Term 1 election: Requested vote from peers 9368997abd4647a194dbef0de676224f (127.2.56.1:37749), 77983fbce2044cc897472ece4586b430 (127.2.56.2:34687)
I20260928 12:25:12.865011 21183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a1b247f3b7f49409e568ef5e2d092a2" candidate_uuid: "bb9109131fb249d99f3ec728b912f961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77983fbce2044cc897472ece4586b430"
I20260928 12:25:12.864966 21052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a1b247f3b7f49409e568ef5e2d092a2" candidate_uuid: "bb9109131fb249d99f3ec728b912f961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9368997abd4647a194dbef0de676224f"
I20260928 12:25:12.865088 21183 raft_consensus.cc:3037] T 2a1b247f3b7f49409e568ef5e2d092a2 P 77983fbce2044cc897472ece4586b430 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:12.865099 21052 raft_consensus.cc:3037] T 2a1b247f3b7f49409e568ef5e2d092a2 P 9368997abd4647a194dbef0de676224f [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:12.865828 21183 raft_consensus.cc:2454] T 2a1b247f3b7f49409e568ef5e2d092a2 P 77983fbce2044cc897472ece4586b430 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb9109131fb249d99f3ec728b912f961 in term 1.
I20260928 12:25:12.865998 21248 leader_election.cc:302] T 2a1b247f3b7f49409e568ef5e2d092a2 P bb9109131fb249d99f3ec728b912f961 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77983fbce2044cc897472ece4586b430, bb9109131fb249d99f3ec728b912f961; no voters:
I20260928 12:25:12.866086 21052 raft_consensus.cc:2454] T 2a1b247f3b7f49409e568ef5e2d092a2 P 9368997abd4647a194dbef0de676224f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb9109131fb249d99f3ec728b912f961 in term 1.
I20260928 12:25:12.866108 21382 raft_consensus.cc:2781] T 2a1b247f3b7f49409e568ef5e2d092a2 P bb9109131fb249d99f3ec728b912f961 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:25:12.866273 21382 raft_consensus.cc:686] T 2a1b247f3b7f49409e568ef5e2d092a2 P bb9109131fb249d99f3ec728b912f961 [term 1 LEADER]: Becoming Leader. State: Replica: bb9109131fb249d99f3ec728b912f961, State: Running, Role: LEADER
I20260928 12:25:12.866364 21382 consensus_queue.cc:237] T 2a1b247f3b7f49409e568ef5e2d092a2 P bb9109131fb249d99f3ec728b912f961 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9109131fb249d99f3ec728b912f961" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 46137 } } peers { permanent_uuid: "9368997abd4647a194dbef0de676224f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37749 } } peers { permanent_uuid: "77983fbce2044cc897472ece4586b430" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34687 } }
I20260928 12:25:12.867129 20910 catalog_manager.cc:5495] T 2a1b247f3b7f49409e568ef5e2d092a2 P bb9109131fb249d99f3ec728b912f961 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb9109131fb249d99f3ec728b912f961 (127.2.56.3). New cstate: current_term: 1 leader_uuid: "bb9109131fb249d99f3ec728b912f961" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9109131fb249d99f3ec728b912f961" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 46137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9368997abd4647a194dbef0de676224f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77983fbce2044cc897472ece4586b430" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34687 } health_report { overall_health: UNKNOWN } } }
I20260928 12:25:12.876673 21371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:12.874604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48182 (local address 127.2.56.3:46137)
0928 12:25:12.874649 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:12.874651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:12.874659 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:25:12.874685 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:12.874686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:12.874701 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:12.874757 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:25:12.874860 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.875314 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.876057 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:12.876257 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:12.876316 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:12.876413 (+ 97us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:25:12.876493 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:25:12.876554 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:12.876592 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 12:25:12.876599 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 12:25:12.878265 21052 raft_consensus.cc:1261] T 2a1b247f3b7f49409e568ef5e2d092a2 P 9368997abd4647a194dbef0de676224f [term 1 FOLLOWER]: Refusing update from remote peer bb9109131fb249d99f3ec728b912f961: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:25:12.878261 21183 raft_consensus.cc:1261] T 2a1b247f3b7f49409e568ef5e2d092a2 P 77983fbce2044cc897472ece4586b430 [term 1 FOLLOWER]: Refusing update from remote peer bb9109131fb249d99f3ec728b912f961: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:25:12.878562 21384 consensus_queue.cc:1035] T 2a1b247f3b7f49409e568ef5e2d092a2 P bb9109131fb249d99f3ec728b912f961 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77983fbce2044cc897472ece4586b430" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 34687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:25:12.878679 21382 consensus_queue.cc:1035] T 2a1b247f3b7f49409e568ef5e2d092a2 P bb9109131fb249d99f3ec728b912f961 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9368997abd4647a194dbef0de676224f" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 37749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:25:12.883085 20910 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:25:12.883167 20910 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:25:12.883810 20910 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1586b2bb30ef4d92b7436770f4c4ad87: Sending DeleteTablet for 3 replicas of tablet 2a1b247f3b7f49409e568ef5e2d092a2
I20260928 12:25:12.884094 21032 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a1b247f3b7f49409e568ef5e2d092a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:25:12 UTC) from {username='slave'} at 127.0.0.1:58518
I20260928 12:25:12.884142 21294 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a1b247f3b7f49409e568ef5e2d092a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:25:12 UTC) from {username='slave'} at 127.0.0.1:48170
I20260928 12:25:12.884191 21163 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a1b247f3b7f49409e568ef5e2d092a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:25:12 UTC) from {username='slave'} at 127.0.0.1:51642
I20260928 12:25:12.884392 21398 tablet_replica.cc:330] T 2a1b247f3b7f49409e568ef5e2d092a2 P bb9109131fb249d99f3ec728b912f961: stopping tablet replica
I20260928 12:25:12.884483 21398 raft_consensus.cc:2229] T 2a1b247f3b7f49409e568ef5e2d092a2 P bb9109131fb249d99f3ec728b912f961 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:25:12.884574 21399 tablet_replica.cc:330] T 2a1b247f3b7f49409e568ef5e2d092a2 P 77983fbce2044cc897472ece4586b430: stopping tablet replica
I20260928 12:25:12.884593 21398 raft_consensus.cc:2258] T 2a1b247f3b7f49409e568ef5e2d092a2 P bb9109131fb249d99f3ec728b912f961 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:25:12.884648 21399 raft_consensus.cc:2229] T 2a1b247f3b7f49409e568ef5e2d092a2 P 77983fbce2044cc897472ece4586b430 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:25:12.884660 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 20970
I20260928 12:25:12.884728 21399 raft_consensus.cc:2258] T 2a1b247f3b7f49409e568ef5e2d092a2 P 77983fbce2044cc897472ece4586b430 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:25:12.887255 21399 ts_tablet_manager.cc:1905] T 2a1b247f3b7f49409e568ef5e2d092a2 P 77983fbce2044cc897472ece4586b430: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:25:12.888594 21398 ts_tablet_manager.cc:1905] T 2a1b247f3b7f49409e568ef5e2d092a2 P bb9109131fb249d99f3ec728b912f961: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:25:12.888729 21399 ts_tablet_manager.cc:1918] T 2a1b247f3b7f49409e568ef5e2d092a2 P 77983fbce2044cc897472ece4586b430: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:25:12.888855 21399 log.cc:1196] T 2a1b247f3b7f49409e568ef5e2d092a2 P 77983fbce2044cc897472ece4586b430: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-1/wal/wals/2a1b247f3b7f49409e568ef5e2d092a2
I20260928 12:25:12.889168 21399 ts_tablet_manager.cc:1939] T 2a1b247f3b7f49409e568ef5e2d092a2 P 77983fbce2044cc897472ece4586b430: Deleting consensus metadata
I20260928 12:25:12.890034 21398 ts_tablet_manager.cc:1918] T 2a1b247f3b7f49409e568ef5e2d092a2 P bb9109131fb249d99f3ec728b912f961: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:25:12.890101 21398 log.cc:1196] T 2a1b247f3b7f49409e568ef5e2d092a2 P bb9109131fb249d99f3ec728b912f961: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598072638673-2272-0/minicluster-data/ts-2/wal/wals/2a1b247f3b7f49409e568ef5e2d092a2
I20260928 12:25:12.890326 21398 ts_tablet_manager.cc:1939] T 2a1b247f3b7f49409e568ef5e2d092a2 P bb9109131fb249d99f3ec728b912f961: Deleting consensus metadata
I20260928 12:25:12.890424 20895 catalog_manager.cc:4841] TS 77983fbce2044cc897472ece4586b430 (127.2.56.2:34687): tablet 2a1b247f3b7f49409e568ef5e2d092a2 (table test-table [id=6bd684ae03384f47b72c71657f61e299]) successfully deleted
I20260928 12:25:12.890686 20895 catalog_manager.cc:4841] TS bb9109131fb249d99f3ec728b912f961 (127.2.56.3:46137): tablet 2a1b247f3b7f49409e568ef5e2d092a2 (table test-table [id=6bd684ae03384f47b72c71657f61e299]) successfully deleted
W20260928 12:25:12.891747 20897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.56.1:37749 (error 108)
I20260928 12:25:12.892203 21372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:12.892079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:37749 (local address 127.0.0.1:58526)
0928 12:25:12.892127 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:12.892172 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.1:37749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 12:25:12.892203 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 21101
W20260928 12:25:12.893693 20897 catalog_manager.cc:4584] TS 9368997abd4647a194dbef0de676224f (127.2.56.1:37749): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2a1b247f3b7f49409e568ef5e2d092a2: Network error: Client connection negotiation failed: client connection to 127.2.56.1:37749: connect: Connection refused (error 111)
I20260928 12:25:12.896354 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 21232
I20260928 12:25:12.900475 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 20879
2026-09-28T12:25:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (620 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:25:12 dist-test-slave-920p krb5kdc[21403](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:25:12 dist-test-slave-920p krb5kdc[21403](info): set up 2 sockets
Sep 28 12:25:12 dist-test-slave-920p krb5kdc[21403](info): commencing operation
krb5kdc: starting...
W20260928 12:25:13.856982 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:25:13 dist-test-slave-920p krb5kdc[21403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:25:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:25:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
I20260928 12:25:13.970402 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:43335
--webserver_interface=127.2.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:46085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:43335
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:25:14.049654 21416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:14.049849 21416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:14.049876 21416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:14.050983 21416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:25:14.051023 21416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:14.051040 21416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:14.051056 21416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:25:14.051072 21416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:25:14.052476 21416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:46085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:43335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:43335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:14.052708 21416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:14.052898 21416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:14.057353 21421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:14.057413 21424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:14.057415 21422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:14.057523 21416 server_base.cc:1034] running on GCE node
I20260928 12:25:14.057729 21416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:14.057991 21416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:14.059131 21416 hybrid_clock.cc:648] HybridClock initialized: now 1790598314059115 us; error 29 us; skew 500 ppm
Sep 28 12:25:14 dist-test-slave-920p krb5kdc[21403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598314, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:14.061674 21416 init.cc:377] Logged in from keytab as kudu/127.2.56.62@KRBTEST.COM (short username kudu)
I20260928 12:25:14.062130 21416 webserver.cc:460] Webserver started at http://127.2.56.62:35657/ using document root <none> and password file <none>
I20260928 12:25:14.062335 21416 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:14.062382 21416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:14.062505 21416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:14.063534 21416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "5c5bea9dcee84d5498e5f2dc90a33671"
format_stamp: "Formatted at 2026-09-28 12:25:14 on dist-test-slave-920p"
server_key: "22e4964921e27b33d8cab3b3f0f734bb"
server_key_iv: "42237e8c6bbc414b9f10034920fe537e"
server_key_version: "encryptionkey@0"
I20260928 12:25:14.063855 21416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "5c5bea9dcee84d5498e5f2dc90a33671"
format_stamp: "Formatted at 2026-09-28 12:25:14 on dist-test-slave-920p"
server_key: "22e4964921e27b33d8cab3b3f0f734bb"
server_key_iv: "42237e8c6bbc414b9f10034920fe537e"
server_key_version: "encryptionkey@0"
I20260928 12:25:14.065171 21416 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 12:25:14.065913 21431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:14.066095 21416 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:25:14.066179 21416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "5c5bea9dcee84d5498e5f2dc90a33671"
format_stamp: "Formatted at 2026-09-28 12:25:14 on dist-test-slave-920p"
server_key: "22e4964921e27b33d8cab3b3f0f734bb"
server_key_iv: "42237e8c6bbc414b9f10034920fe537e"
server_key_version: "encryptionkey@0"
I20260928 12:25:14.066242 21416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:14.074520 21416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:14.076165 21416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:14.076287 21416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:14.079854 21416 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:43335
I20260928 12:25:14.079895 21483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:43335 every 8 connection(s)
I20260928 12:25:14.080201 21416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:25:14.081032 21484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:14.083209 21484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c5bea9dcee84d5498e5f2dc90a33671: Bootstrap starting.
I20260928 12:25:14.083750 21484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c5bea9dcee84d5498e5f2dc90a33671: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:14.083998 21484 log.cc:824] T 00000000000000000000000000000000 P 5c5bea9dcee84d5498e5f2dc90a33671: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:14.084627 21484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c5bea9dcee84d5498e5f2dc90a33671: No bootstrap required, opened a new log
I20260928 12:25:14.084650 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 21416
I20260928 12:25:14.084743 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:25:14.084913 2272 external_mini_cluster.cc:1428] Setting key 08cebc630bc85119f2e09999dadd1e91
I20260928 12:25:14.086577 21484 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c5bea9dcee84d5498e5f2dc90a33671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c5bea9dcee84d5498e5f2dc90a33671" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43335 } }
I20260928 12:25:14.086758 21484 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c5bea9dcee84d5498e5f2dc90a33671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:14.086795 21484 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c5bea9dcee84d5498e5f2dc90a33671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c5bea9dcee84d5498e5f2dc90a33671, State: Initialized, Role: FOLLOWER
I20260928 12:25:14.086933 21484 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c5bea9dcee84d5498e5f2dc90a33671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c5bea9dcee84d5498e5f2dc90a33671" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43335 } }
I20260928 12:25:14.086997 21484 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c5bea9dcee84d5498e5f2dc90a33671 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:25:14.087015 21484 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c5bea9dcee84d5498e5f2dc90a33671 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:25:14.087047 21484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c5bea9dcee84d5498e5f2dc90a33671 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:14.088039 21484 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c5bea9dcee84d5498e5f2dc90a33671 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c5bea9dcee84d5498e5f2dc90a33671" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43335 } }
I20260928 12:25:14.088145 21484 leader_election.cc:302] T 00000000000000000000000000000000 P 5c5bea9dcee84d5498e5f2dc90a33671 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c5bea9dcee84d5498e5f2dc90a33671; no voters:
I20260928 12:25:14.088289 21484 leader_election.cc:288] T 00000000000000000000000000000000 P 5c5bea9dcee84d5498e5f2dc90a33671 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:25:14.088323 21489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c5bea9dcee84d5498e5f2dc90a33671 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:25:14.088479 21489 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c5bea9dcee84d5498e5f2dc90a33671 [term 1 LEADER]: Becoming Leader. State: Replica: 5c5bea9dcee84d5498e5f2dc90a33671, State: Running, Role: LEADER
I20260928 12:25:14.088514 21484 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c5bea9dcee84d5498e5f2dc90a33671 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:25:14.088589 21489 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c5bea9dcee84d5498e5f2dc90a33671 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c5bea9dcee84d5498e5f2dc90a33671" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43335 } }
I20260928 12:25:14.088914 21489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c5bea9dcee84d5498e5f2dc90a33671 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c5bea9dcee84d5498e5f2dc90a33671. Latest consensus state: current_term: 1 leader_uuid: "5c5bea9dcee84d5498e5f2dc90a33671" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c5bea9dcee84d5498e5f2dc90a33671" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43335 } } }
I20260928 12:25:14.088943 21490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c5bea9dcee84d5498e5f2dc90a33671 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c5bea9dcee84d5498e5f2dc90a33671" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c5bea9dcee84d5498e5f2dc90a33671" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 43335 } } }
I20260928 12:25:14.088984 21489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c5bea9dcee84d5498e5f2dc90a33671 [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:14.089028 21490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c5bea9dcee84d5498e5f2dc90a33671 [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:14.089221 21497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:25:14.089757 21497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:25:14.091450 21497 catalog_manager.cc:1347] Generated new cluster ID: 4eb7b7b1301847d5ae524cfe47d6da03
I20260928 12:25:14.091492 21497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:25:14 dist-test-slave-920p krb5kdc[21403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:25:14.097430 21488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:14.085457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42916 (local address 127.2.56.62:43335)
0928 12:25:14.085656 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:14.085665 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:14.085698 (+ 33us) server_negotiation.cc:407] Connection header received
0928 12:25:14.086548 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:14.086557 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:14.086618 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:14.086759 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:14.087436 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.088769 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.089833 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.090430 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.090532 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:14.093329 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:14.093333 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:14.093334 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:14.093346 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:14.095278 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:14.095669 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:14.095672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:14.095672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:14.095808 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:14.095982 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:14.095983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:14.095983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:14.096308 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:14.096950 (+ 642us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:14.097024 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 12:25:14.097180 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":98,"threads_started":1}
I20260928 12:25:14.101672 21497 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:25:14.102383 21497 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:25:14.105477 21497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c5bea9dcee84d5498e5f2dc90a33671: Generated new TSK 0
I20260928 12:25:14.105628 21497 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab.
Entry for principal kudu/127.2.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab.
I20260928 12:25:14.130152 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.1:0
--local_ip_for_outbound_sockets=127.2.56.1
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:43335
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:46085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:25:14.208315 21510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:14.208498 21510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:14.208518 21510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:14.209662 21510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:14.209699 21510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:14.209724 21510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.1
I20260928 12:25:14.211103 21510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:46085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.1.keytab
--principal=kudu/127.2.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:43335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.1
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:14.211302 21510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:14.211508 21510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:14.216092 21518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:14.216190 21510 server_base.cc:1034] running on GCE node
W20260928 12:25:14.216100 21515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:14.216099 21516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:14.216496 21510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:14.216717 21510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:14.217846 21510 hybrid_clock.cc:648] HybridClock initialized: now 1790598314217834 us; error 24 us; skew 500 ppm
Sep 28 12:25:14 dist-test-slave-920p krb5kdc[21403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598314, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:14.220079 21510 init.cc:377] Logged in from keytab as kudu/127.2.56.1@KRBTEST.COM (short username kudu)
I20260928 12:25:14.220459 21510 webserver.cc:460] Webserver started at http://127.2.56.1:36155/ using document root <none> and password file <none>
I20260928 12:25:14.220659 21510 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:14.220705 21510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:14.220818 21510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:14.221827 21510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-0/data/instance:
uuid: "c5aad6f357ab4db8a937eb0f65bc806a"
format_stamp: "Formatted at 2026-09-28 12:25:14 on dist-test-slave-920p"
server_key: "a5e11bcb25554428cd85b248fe871ebd"
server_key_iv: "3f8c8752f86338cef94a1a19554405b7"
server_key_version: "encryptionkey@0"
I20260928 12:25:14.222138 21510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance:
uuid: "c5aad6f357ab4db8a937eb0f65bc806a"
format_stamp: "Formatted at 2026-09-28 12:25:14 on dist-test-slave-920p"
server_key: "a5e11bcb25554428cd85b248fe871ebd"
server_key_iv: "3f8c8752f86338cef94a1a19554405b7"
server_key_version: "encryptionkey@0"
I20260928 12:25:14.223280 21510 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:25:14.223992 21525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:14.224156 21510 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:25:14.224231 21510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-0/wal
uuid: "c5aad6f357ab4db8a937eb0f65bc806a"
format_stamp: "Formatted at 2026-09-28 12:25:14 on dist-test-slave-920p"
server_key: "a5e11bcb25554428cd85b248fe871ebd"
server_key_iv: "3f8c8752f86338cef94a1a19554405b7"
server_key_version: "encryptionkey@0"
I20260928 12:25:14.224292 21510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:14.254757 21510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:14.256505 21510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:14.256628 21510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:14.256831 21510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:25:14.257107 21510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:25:14.257139 21510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:14.257177 21510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:25:14.257201 21510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:14.262298 21510 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.1:44023
I20260928 12:25:14.262353 21638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.1:44023 every 8 connection(s)
I20260928 12:25:14.262637 21510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
I20260928 12:25:14.264600 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 21510
I20260928 12:25:14.264693 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance
I20260928 12:25:14.264896 2272 external_mini_cluster.cc:1428] Setting key 8fcb31e10f7f6e02e7af9862d4ad3497
Sep 28 12:25:14 dist-test-slave-920p krb5kdc[21403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598314, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.1@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:25:14.271940 21488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:14.263296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:47601 (local address 127.2.56.62:43335)
0928 12:25:14.263370 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:14.263373 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:14.263515 (+ 142us) server_negotiation.cc:407] Connection header received
0928 12:25:14.263956 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:14.263957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:14.263970 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:14.264017 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:14.265115 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.265758 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.266789 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.266998 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.267051 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:14.269114 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:14.269116 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:14.269116 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:14.269126 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:14.270639 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:14.270996 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:14.270997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:14.270997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:14.271071 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:14.271266 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:14.271267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:14.271267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:14.271399 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:14.271467 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:14.271759 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 12:25:14.271843 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 12:25:14.272003 21641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:14.263266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43335 (local address 127.2.56.1:47601)
0928 12:25:14.263406 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:14.263440 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:14.263883 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:14.264090 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:14.264093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:14.264749 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:14.265025 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:14.265030 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.265836 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.265837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:14.266682 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:14.266686 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.267076 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.267076 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:14.267210 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:14.267678 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:25:14.267688 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:14.269021 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:14.270713 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:14.270719 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:14.270721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:14.270927 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:14.271129 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:14.271130 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:14.271130 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:14.271213 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:14.271485 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:14.271491 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:14.271625 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 12:25:14.271718 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 12:25:14.271838 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":79,"threads_started":1}
I20260928 12:25:14.272533 21639 heartbeater.cc:347] Connected to a master server at 127.2.56.62:43335
I20260928 12:25:14.272620 21639 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.2.56.2@KRBTEST.COM; defaulting to no policy
I20260928 12:25:14.272794 21639 heartbeater.cc:511] Master 127.2.56.62:43335 requested a full tablet report, sending...
I20260928 12:25:14.273316 21448 ts_manager.cc:194] Registered new tserver with Master: c5aad6f357ab4db8a937eb0f65bc806a (127.2.56.1:44023)
I20260928 12:25:14.274035 21448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.1@KRBTEST.COM'} at 127.2.56.1:47601
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab.
Entry for principal kudu/127.2.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab.
I20260928 12:25:14.287703 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.2:0
--local_ip_for_outbound_sockets=127.2.56.2
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:43335
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:46085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:25:14.366427 21644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:14.366614 21644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:14.366641 21644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:14.367756 21644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:14.367794 21644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:14.367822 21644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.2
I20260928 12:25:14.369240 21644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:46085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.2.keytab
--principal=kudu/127.2.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:43335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.2
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:14.369477 21644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:14.369673 21644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:14.374311 21652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:14.374320 21644 server_base.cc:1034] running on GCE node
W20260928 12:25:14.374420 21650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:14.374560 21649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:14.374778 21644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:14.374962 21644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:14.376088 21644 hybrid_clock.cc:648] HybridClock initialized: now 1790598314376078 us; error 27 us; skew 500 ppm
Sep 28 12:25:14 dist-test-slave-920p krb5kdc[21403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598314, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:14.378453 21644 init.cc:377] Logged in from keytab as kudu/127.2.56.2@KRBTEST.COM (short username kudu)
I20260928 12:25:14.378857 21644 webserver.cc:460] Webserver started at http://127.2.56.2:33125/ using document root <none> and password file <none>
I20260928 12:25:14.379045 21644 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:14.379091 21644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:14.379209 21644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:14.380184 21644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-1/data/instance:
uuid: "6a4074e801f844c58f76cfc8c9415b71"
format_stamp: "Formatted at 2026-09-28 12:25:14 on dist-test-slave-920p"
server_key: "2913706b8e3f4f2604aa8850bedee629"
server_key_iv: "5586d7bfc0e22c3d8ccfd6f184f8d484"
server_key_version: "encryptionkey@0"
I20260928 12:25:14.380494 21644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance:
uuid: "6a4074e801f844c58f76cfc8c9415b71"
format_stamp: "Formatted at 2026-09-28 12:25:14 on dist-test-slave-920p"
server_key: "2913706b8e3f4f2604aa8850bedee629"
server_key_iv: "5586d7bfc0e22c3d8ccfd6f184f8d484"
server_key_version: "encryptionkey@0"
I20260928 12:25:14.381723 21644 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:25:14.382391 21659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:14.382526 21644 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:25:14.382609 21644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-1/wal
uuid: "6a4074e801f844c58f76cfc8c9415b71"
format_stamp: "Formatted at 2026-09-28 12:25:14 on dist-test-slave-920p"
server_key: "2913706b8e3f4f2604aa8850bedee629"
server_key_iv: "5586d7bfc0e22c3d8ccfd6f184f8d484"
server_key_version: "encryptionkey@0"
I20260928 12:25:14.382668 21644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:14.393985 21644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:14.395952 21644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:14.396080 21644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:14.396288 21644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:25:14.396572 21644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:25:14.396606 21644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:14.396633 21644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:25:14.396649 21644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:14.401965 21644 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.2:37585
I20260928 12:25:14.402027 21772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.2:37585 every 8 connection(s)
I20260928 12:25:14.402284 21644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
Sep 28 12:25:14 dist-test-slave-920p krb5kdc[21403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598314, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.2@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:25:14.410537 21488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:14.402885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:48919 (local address 127.2.56.62:43335)
0928 12:25:14.402932 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:14.402934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:14.403150 (+ 216us) server_negotiation.cc:407] Connection header received
0928 12:25:14.403650 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:14.403651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:14.403663 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:14.403710 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:14.404591 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.405086 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.406080 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.406273 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.406329 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:14.408215 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:14.408217 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:14.408217 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:14.408224 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:14.409349 (+ 1125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:14.409695 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:14.409696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:14.409697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:14.409748 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:14.409950 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:14.409951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:14.409951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:14.410058 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:14.410114 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:14.410362 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 12:25:14.410447 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:25:14.410686 21775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:14.402850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43335 (local address 127.2.56.2:48919)
0928 12:25:14.403014 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:14.403047 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:14.403548 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:14.403785 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:14.403788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:14.404323 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:14.404504 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:14.404509 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.405166 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.405167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:14.405985 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:14.405988 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.406340 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.406341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:14.406460 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:14.406851 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:25:14.406858 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:14.408130 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:14.409421 (+ 1291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:14.409428 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:14.409430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:14.409632 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:14.409800 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:14.409801 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:14.409801 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:14.409898 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:14.410119 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:14.410125 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:14.410251 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 12:25:14.410336 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 12:25:14.410482 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":86,"threads_started":1}
I20260928 12:25:14.411167 21773 heartbeater.cc:347] Connected to a master server at 127.2.56.62:43335
I20260928 12:25:14.411248 21773 heartbeater.cc:464] Registering TS with master...
I20260928 12:25:14.411381 21773 heartbeater.cc:511] Master 127.2.56.62:43335 requested a full tablet report, sending...
I20260928 12:25:14.411718 21448 ts_manager.cc:194] Registered new tserver with Master: 6a4074e801f844c58f76cfc8c9415b71 (127.2.56.2:37585)
I20260928 12:25:14.412011 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 21644
I20260928 12:25:14.412106 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance
I20260928 12:25:14.412173 21448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.2@KRBTEST.COM'} at 127.2.56.2:48919
I20260928 12:25:14.412289 2272 external_mini_cluster.cc:1428] Setting key 03395a41a415650c2e80a27a94f4cc03
WARNING: no policy specified for kudu/127.2.56.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab.
Entry for principal kudu/127.2.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab.
I20260928 12:25:14.434031 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.3:0
--local_ip_for_outbound_sockets=127.2.56.3
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:43335
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:46085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:25:14.511355 21778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:14.511557 21778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:14.511592 21778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:14.512705 21778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:14.512743 21778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:14.512777 21778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.3
I20260928 12:25:14.514259 21778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:46085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.3.keytab
--principal=kudu/127.2.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:43335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.3
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:14.514487 21778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:14.514686 21778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:14.519193 21784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:14.519205 21783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:14.519205 21786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:14.519419 21778 server_base.cc:1034] running on GCE node
I20260928 12:25:14.519601 21778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:14.519804 21778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:14.520941 21778 hybrid_clock.cc:648] HybridClock initialized: now 1790598314520928 us; error 25 us; skew 500 ppm
Sep 28 12:25:14 dist-test-slave-920p krb5kdc[21403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598314, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:14.523478 21778 init.cc:377] Logged in from keytab as kudu/127.2.56.3@KRBTEST.COM (short username kudu)
I20260928 12:25:14.523901 21778 webserver.cc:460] Webserver started at http://127.2.56.3:33085/ using document root <none> and password file <none>
I20260928 12:25:14.524135 21778 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:14.524184 21778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:14.524307 21778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:14.525367 21778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-2/data/instance:
uuid: "ce33b920ea03429892c0474c2312783a"
format_stamp: "Formatted at 2026-09-28 12:25:14 on dist-test-slave-920p"
server_key: "0fb9fc2e3f22771b2404da8cd5e52691"
server_key_iv: "47f62333afecc7908a91e0ea510cb0b2"
server_key_version: "encryptionkey@0"
I20260928 12:25:14.525688 21778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance:
uuid: "ce33b920ea03429892c0474c2312783a"
format_stamp: "Formatted at 2026-09-28 12:25:14 on dist-test-slave-920p"
server_key: "0fb9fc2e3f22771b2404da8cd5e52691"
server_key_iv: "47f62333afecc7908a91e0ea510cb0b2"
server_key_version: "encryptionkey@0"
I20260928 12:25:14.526932 21778 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:25:14.527608 21793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:14.527750 21778 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:25:14.527828 21778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-2/wal
uuid: "ce33b920ea03429892c0474c2312783a"
format_stamp: "Formatted at 2026-09-28 12:25:14 on dist-test-slave-920p"
server_key: "0fb9fc2e3f22771b2404da8cd5e52691"
server_key_iv: "47f62333afecc7908a91e0ea510cb0b2"
server_key_version: "encryptionkey@0"
I20260928 12:25:14.527886 21778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:14.543687 21778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:14.545665 21778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:14.545800 21778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:14.546025 21778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:25:14.546316 21778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:25:14.546348 21778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:14.546377 21778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:25:14.546396 21778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:14.551651 21778 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.3:32823
I20260928 12:25:14.551708 21906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.3:32823 every 8 connection(s)
I20260928 12:25:14.551990 21778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
Sep 28 12:25:14 dist-test-slave-920p krb5kdc[21403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598314, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.3@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:25:14.558030 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 21778
I20260928 12:25:14.558146 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance
I20260928 12:25:14.558332 2272 external_mini_cluster.cc:1428] Setting key 2593d60415085d310e2ef0a6ffcf0cbb
I20260928 12:25:14.560465 21488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:14.552690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:37361 (local address 127.2.56.62:43335)
0928 12:25:14.552759 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:14.552761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:14.552940 (+ 179us) server_negotiation.cc:407] Connection header received
0928 12:25:14.553417 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:14.553418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:14.553427 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:14.553476 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:14.554368 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.554852 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.555798 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.555995 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.556047 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:14.558071 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:14.558072 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:14.558073 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:14.558078 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:14.559499 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:14.559818 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:14.559818 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:14.559818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:14.559861 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:14.560018 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:14.560018 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:14.560019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:14.560105 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:14.560147 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:14.560350 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 12:25:14.560404 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 12:25:14.560655 21909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:14.552657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:43335 (local address 127.2.56.3:37361)
0928 12:25:14.552829 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:14.552865 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:14.553345 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:14.553537 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:14.553540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:14.554112 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:14.554300 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:14.554305 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.554926 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.554927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:14.555715 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:14.555717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.556085 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.556086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:14.556202 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:14.556660 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:25:14.556667 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:14.557980 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:14.559564 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:14.559569 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:14.559571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:14.559764 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:14.559895 (+ 131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:14.559895 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:14.559896 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:14.559979 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:14.560147 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:14.560150 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:14.560256 (+ 106us) client_negotiation.cc:769] Sending connection context
0928 12:25:14.560331 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 12:25:14.560441 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":75,"threads_started":1}
I20260928 12:25:14.561039 21907 heartbeater.cc:347] Connected to a master server at 127.2.56.62:43335
I20260928 12:25:14.561120 21907 heartbeater.cc:464] Registering TS with master...
I20260928 12:25:14.561312 21907 heartbeater.cc:511] Master 127.2.56.62:43335 requested a full tablet report, sending...
I20260928 12:25:14.561635 21448 ts_manager.cc:194] Registered new tserver with Master: ce33b920ea03429892c0474c2312783a (127.2.56.3:32823)
I20260928 12:25:14.561954 21448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.56.3@KRBTEST.COM'} at 127.2.56.3:37361
I20260928 12:25:14.569902 2272 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:25:14.576236 21488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:14.571175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42920 (local address 127.2.56.62:43335)
0928 12:25:14.571216 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:14.571218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:14.571300 (+ 82us) server_negotiation.cc:407] Connection header received
0928 12:25:14.571350 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:14.571350 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:14.571358 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:14.571404 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:14.571947 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.572429 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.573075 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.573245 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.573309 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:14.574327 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:14.574329 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:14.574329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:14.574334 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:14.575485 (+ 1151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:14.575724 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:14.575725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:14.575725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:14.575765 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:14.575870 (+ 105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:14.575870 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:14.575870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:14.575987 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:14.576071 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:14.576124 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 12:25:14.576177 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:25:14.577692 21448 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:25:14.579567 21448 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:25:14.585596 21915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:14.581636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:37585 (local address 127.0.0.1:47262)
0928 12:25:14.581984 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:14.581991 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:14.582020 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:14.582296 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:14.582299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:14.582301 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:25:14.582365 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:14.582367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.583672 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.583673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:14.584438 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:14.584440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.585304 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.585305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:14.585383 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:14.585396 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 12:25:14.585466 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 12:25:14.585516 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":32,"thread_start_us":51,"threads_started":1}
I20260928 12:25:14.585605 21917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:14.581729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47262 (local address 127.2.56.2:37585)
0928 12:25:14.582120 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:14.582126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:14.582137 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:25:14.582177 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:14.582180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:14.582197 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:14.582275 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:25:14.582450 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.583564 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.584532 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.585212 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.585303 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:14.585361 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:14.585487 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 12:25:14.585526 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":100,"threads_started":1}
I20260928 12:25:14.585855 21919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:14.581913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36328 (local address 127.2.56.3:32823)
0928 12:25:14.582088 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:14.582092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:14.582228 (+ 136us) server_negotiation.cc:407] Connection header received
0928 12:25:14.582394 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:14.582396 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:14.582412 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:14.582461 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:25:14.582659 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.583328 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.584698 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.585450 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.585509 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:14.585557 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:14.585750 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 12:25:14.585795 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
I20260928 12:25:14.586095 21918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:14.581816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:32823 (local address 127.0.0.1:36328)
0928 12:25:14.582160 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:14.582170 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:14.582254 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:14.582555 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:14.582556 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:14.582557 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:25:14.582607 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:14.582609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.583413 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.583413 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:14.584624 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:14.584627 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.585590 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.585591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:14.585647 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:14.585651 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:25:14.585953 (+ 302us) client_negotiation.cc:240] Negotiation successful
0928 12:25:14.586032 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":30,"thread_start_us":61,"threads_started":1}
I20260928 12:25:14.586390 21916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:14.581636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:44023 (local address 127.0.0.1:58652)
0928 12:25:14.581804 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:14.581845 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:14.581909 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:14.582356 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:14.582357 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:14.582358 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:25:14.582387 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:14.582388 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.583657 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.583658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:14.584926 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:14.584928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.586173 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.586174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:14.586240 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:14.586243 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:25:14.586286 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 12:25:14.586321 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":56,"threads_started":1}
I20260928 12:25:14.586421 21707 tablet_service.cc:1467] Processing CreateTablet for tablet 526dbf271b144fa78982f0b984472199 (DEFAULT_TABLE table=test-table [id=c795dfda228942bdac756dc55dbcaaa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:25:14.586761 21707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 526dbf271b144fa78982f0b984472199. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:14.586865 21920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:14.581965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58652 (local address 127.2.56.1:44023)
0928 12:25:14.582161 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:14.582165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:14.582172 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:25:14.582204 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:14.582205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:14.582216 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:14.582278 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:25:14.582457 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.583562 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.585003 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.585629 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.586079 (+ 450us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:14.586144 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:14.586746 (+ 602us) server_negotiation.cc:299] Negotiation successful
0928 12:25:14.586789 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":99,"threads_started":1}
I20260928 12:25:14.586802 21841 tablet_service.cc:1467] Processing CreateTablet for tablet 526dbf271b144fa78982f0b984472199 (DEFAULT_TABLE table=test-table [id=c795dfda228942bdac756dc55dbcaaa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:25:14.587085 21841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 526dbf271b144fa78982f0b984472199. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:14.589424 21921 tablet_bootstrap.cc:492] T 526dbf271b144fa78982f0b984472199 P ce33b920ea03429892c0474c2312783a: Bootstrap starting.
I20260928 12:25:14.589668 21922 tablet_bootstrap.cc:492] T 526dbf271b144fa78982f0b984472199 P 6a4074e801f844c58f76cfc8c9415b71: Bootstrap starting.
I20260928 12:25:14.590121 21921 tablet_bootstrap.cc:654] T 526dbf271b144fa78982f0b984472199 P ce33b920ea03429892c0474c2312783a: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:14.590101 21573 tablet_service.cc:1467] Processing CreateTablet for tablet 526dbf271b144fa78982f0b984472199 (DEFAULT_TABLE table=test-table [id=c795dfda228942bdac756dc55dbcaaa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:25:14.590324 21922 tablet_bootstrap.cc:654] T 526dbf271b144fa78982f0b984472199 P 6a4074e801f844c58f76cfc8c9415b71: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:14.590441 21573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 526dbf271b144fa78982f0b984472199. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:14.590629 21921 log.cc:824] T 526dbf271b144fa78982f0b984472199 P ce33b920ea03429892c0474c2312783a: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:14.590853 21922 log.cc:824] T 526dbf271b144fa78982f0b984472199 P 6a4074e801f844c58f76cfc8c9415b71: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:14.591284 21921 tablet_bootstrap.cc:492] T 526dbf271b144fa78982f0b984472199 P ce33b920ea03429892c0474c2312783a: No bootstrap required, opened a new log
I20260928 12:25:14.591349 21921 ts_tablet_manager.cc:1397] T 526dbf271b144fa78982f0b984472199 P ce33b920ea03429892c0474c2312783a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:25:14.591530 21922 tablet_bootstrap.cc:492] T 526dbf271b144fa78982f0b984472199 P 6a4074e801f844c58f76cfc8c9415b71: No bootstrap required, opened a new log
I20260928 12:25:14.591596 21922 ts_tablet_manager.cc:1397] T 526dbf271b144fa78982f0b984472199 P 6a4074e801f844c58f76cfc8c9415b71: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:25:14.592684 21921 raft_consensus.cc:348] T 526dbf271b144fa78982f0b984472199 P ce33b920ea03429892c0474c2312783a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5aad6f357ab4db8a937eb0f65bc806a" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44023 } } peers { permanent_uuid: "ce33b920ea03429892c0474c2312783a" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 32823 } } peers { permanent_uuid: "6a4074e801f844c58f76cfc8c9415b71" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 37585 } }
I20260928 12:25:14.592837 21921 raft_consensus.cc:374] T 526dbf271b144fa78982f0b984472199 P ce33b920ea03429892c0474c2312783a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:14.592876 21921 raft_consensus.cc:729] T 526dbf271b144fa78982f0b984472199 P ce33b920ea03429892c0474c2312783a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce33b920ea03429892c0474c2312783a, State: Initialized, Role: FOLLOWER
I20260928 12:25:14.593001 21921 consensus_queue.cc:260] T 526dbf271b144fa78982f0b984472199 P ce33b920ea03429892c0474c2312783a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5aad6f357ab4db8a937eb0f65bc806a" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44023 } } peers { permanent_uuid: "ce33b920ea03429892c0474c2312783a" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 32823 } } peers { permanent_uuid: "6a4074e801f844c58f76cfc8c9415b71" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 37585 } }
I20260928 12:25:14.593242 21921 ts_tablet_manager.cc:1428] T 526dbf271b144fa78982f0b984472199 P ce33b920ea03429892c0474c2312783a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:25:14.593369 21907 heartbeater.cc:503] Master 127.2.56.62:43335 was elected leader, sending a full tablet report...
I20260928 12:25:14.593516 21922 raft_consensus.cc:348] T 526dbf271b144fa78982f0b984472199 P 6a4074e801f844c58f76cfc8c9415b71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5aad6f357ab4db8a937eb0f65bc806a" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44023 } } peers { permanent_uuid: "ce33b920ea03429892c0474c2312783a" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 32823 } } peers { permanent_uuid: "6a4074e801f844c58f76cfc8c9415b71" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 37585 } }
I20260928 12:25:14.593683 21922 raft_consensus.cc:374] T 526dbf271b144fa78982f0b984472199 P 6a4074e801f844c58f76cfc8c9415b71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:14.593721 21922 raft_consensus.cc:729] T 526dbf271b144fa78982f0b984472199 P 6a4074e801f844c58f76cfc8c9415b71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a4074e801f844c58f76cfc8c9415b71, State: Initialized, Role: FOLLOWER
I20260928 12:25:14.593818 21922 consensus_queue.cc:260] T 526dbf271b144fa78982f0b984472199 P 6a4074e801f844c58f76cfc8c9415b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5aad6f357ab4db8a937eb0f65bc806a" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44023 } } peers { permanent_uuid: "ce33b920ea03429892c0474c2312783a" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 32823 } } peers { permanent_uuid: "6a4074e801f844c58f76cfc8c9415b71" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 37585 } }
I20260928 12:25:14.594063 21922 ts_tablet_manager.cc:1428] T 526dbf271b144fa78982f0b984472199 P 6a4074e801f844c58f76cfc8c9415b71: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 12:25:14.594099 21926 tablet_bootstrap.cc:492] T 526dbf271b144fa78982f0b984472199 P c5aad6f357ab4db8a937eb0f65bc806a: Bootstrap starting.
I20260928 12:25:14.594172 21773 heartbeater.cc:503] Master 127.2.56.62:43335 was elected leader, sending a full tablet report...
I20260928 12:25:14.594717 21926 tablet_bootstrap.cc:654] T 526dbf271b144fa78982f0b984472199 P c5aad6f357ab4db8a937eb0f65bc806a: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:14.595199 21926 log.cc:824] T 526dbf271b144fa78982f0b984472199 P c5aad6f357ab4db8a937eb0f65bc806a: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:14.595737 21926 tablet_bootstrap.cc:492] T 526dbf271b144fa78982f0b984472199 P c5aad6f357ab4db8a937eb0f65bc806a: No bootstrap required, opened a new log
I20260928 12:25:14.595782 21926 ts_tablet_manager.cc:1397] T 526dbf271b144fa78982f0b984472199 P c5aad6f357ab4db8a937eb0f65bc806a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:25:14.597016 21926 raft_consensus.cc:348] T 526dbf271b144fa78982f0b984472199 P c5aad6f357ab4db8a937eb0f65bc806a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5aad6f357ab4db8a937eb0f65bc806a" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44023 } } peers { permanent_uuid: "ce33b920ea03429892c0474c2312783a" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 32823 } } peers { permanent_uuid: "6a4074e801f844c58f76cfc8c9415b71" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 37585 } }
I20260928 12:25:14.597137 21926 raft_consensus.cc:374] T 526dbf271b144fa78982f0b984472199 P c5aad6f357ab4db8a937eb0f65bc806a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:14.597157 21926 raft_consensus.cc:729] T 526dbf271b144fa78982f0b984472199 P c5aad6f357ab4db8a937eb0f65bc806a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5aad6f357ab4db8a937eb0f65bc806a, State: Initialized, Role: FOLLOWER
I20260928 12:25:14.597244 21926 consensus_queue.cc:260] T 526dbf271b144fa78982f0b984472199 P c5aad6f357ab4db8a937eb0f65bc806a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5aad6f357ab4db8a937eb0f65bc806a" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44023 } } peers { permanent_uuid: "ce33b920ea03429892c0474c2312783a" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 32823 } } peers { permanent_uuid: "6a4074e801f844c58f76cfc8c9415b71" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 37585 } }
I20260928 12:25:14.597492 21926 ts_tablet_manager.cc:1428] T 526dbf271b144fa78982f0b984472199 P c5aad6f357ab4db8a937eb0f65bc806a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:25:14.597544 21639 heartbeater.cc:503] Master 127.2.56.62:43335 was elected leader, sending a full tablet report...
W20260928 12:25:14.652894 21774 tablet.cc:2369] T 526dbf271b144fa78982f0b984472199 P 6a4074e801f844c58f76cfc8c9415b71: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:25:14.699842 21929 raft_consensus.cc:482] T 526dbf271b144fa78982f0b984472199 P c5aad6f357ab4db8a937eb0f65bc806a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:25:14.699982 21929 raft_consensus.cc:504] T 526dbf271b144fa78982f0b984472199 P c5aad6f357ab4db8a937eb0f65bc806a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5aad6f357ab4db8a937eb0f65bc806a" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44023 } } peers { permanent_uuid: "ce33b920ea03429892c0474c2312783a" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 32823 } } peers { permanent_uuid: "6a4074e801f844c58f76cfc8c9415b71" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 37585 } }
I20260928 12:25:14.700464 21927 raft_consensus.cc:482] T 526dbf271b144fa78982f0b984472199 P 6a4074e801f844c58f76cfc8c9415b71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:25:14.700570 21927 raft_consensus.cc:504] T 526dbf271b144fa78982f0b984472199 P 6a4074e801f844c58f76cfc8c9415b71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5aad6f357ab4db8a937eb0f65bc806a" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44023 } } peers { permanent_uuid: "ce33b920ea03429892c0474c2312783a" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 32823 } } peers { permanent_uuid: "6a4074e801f844c58f76cfc8c9415b71" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 37585 } }
I20260928 12:25:14.700660 21929 leader_election.cc:288] T 526dbf271b144fa78982f0b984472199 P c5aad6f357ab4db8a937eb0f65bc806a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce33b920ea03429892c0474c2312783a (127.2.56.3:32823), 6a4074e801f844c58f76cfc8c9415b71 (127.2.56.2:37585)
I20260928 12:25:14.700891 21927 leader_election.cc:288] T 526dbf271b144fa78982f0b984472199 P 6a4074e801f844c58f76cfc8c9415b71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5aad6f357ab4db8a937eb0f65bc806a (127.2.56.1:44023), ce33b920ea03429892c0474c2312783a (127.2.56.3:32823)
I20260928 12:25:14.704082 21917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:14.700571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:54265 (local address 127.2.56.2:37585)
0928 12:25:14.700844 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:14.700846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:14.700856 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:25:14.700879 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:14.700880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:14.700890 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:14.700937 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:25:14.701143 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.701823 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.703269 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.703746 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.703782 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:14.703793 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:14.703984 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 12:25:14.704018 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250}
I20260928 12:25:14.704130 21641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:14.700523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:32823 (local address 127.2.56.1:59449)
0928 12:25:14.700628 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:14.700640 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:14.700704 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:14.701005 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:14.701006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:14.701008 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:25:14.701068 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:14.701070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.701964 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.701965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:14.703182 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:14.703184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.703906 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.703907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:14.703976 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:14.703979 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:25:14.704019 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 12:25:14.704059 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":32}
I20260928 12:25:14.704310 21930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:14.700523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:37585 (local address 127.2.56.1:54265)
0928 12:25:14.700767 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:14.700776 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:14.700822 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:14.700984 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:14.700985 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:14.700987 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:25:14.701058 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:14.701061 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.701903 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.701904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:14.703177 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:14.703180 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.703844 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.703845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:14.703911 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:14.703916 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:25:14.704194 (+ 278us) client_negotiation.cc:240] Negotiation successful
0928 12:25:14.704237 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":79,"thread_start_us":87,"threads_started":1}
I20260928 12:25:14.704370 21919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:14.700789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:59449 (local address 127.2.56.3:32823)
0928 12:25:14.700859 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:14.700861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:14.700869 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:25:14.700895 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:14.700896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:14.700907 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:14.700957 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:25:14.701143 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.701887 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.703267 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.703822 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.703873 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:14.703882 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:14.704273 (+ 391us) server_negotiation.cc:299] Negotiation successful
0928 12:25:14.704310 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 12:25:14.704574 21727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526dbf271b144fa78982f0b984472199" candidate_uuid: "c5aad6f357ab4db8a937eb0f65bc806a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a4074e801f844c58f76cfc8c9415b71" is_pre_election: true
I20260928 12:25:14.704602 21861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526dbf271b144fa78982f0b984472199" candidate_uuid: "c5aad6f357ab4db8a937eb0f65bc806a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce33b920ea03429892c0474c2312783a" is_pre_election: true
I20260928 12:25:14.704686 21727 raft_consensus.cc:2454] T 526dbf271b144fa78982f0b984472199 P 6a4074e801f844c58f76cfc8c9415b71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5aad6f357ab4db8a937eb0f65bc806a in term 0.
I20260928 12:25:14.704715 21861 raft_consensus.cc:2454] T 526dbf271b144fa78982f0b984472199 P ce33b920ea03429892c0474c2312783a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5aad6f357ab4db8a937eb0f65bc806a in term 0.
I20260928 12:25:14.704790 21775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:14.700996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:44023 (local address 127.2.56.2:52853)
0928 12:25:14.701242 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:14.701251 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:14.701313 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:14.701518 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:14.701519 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:14.701520 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:25:14.701576 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:14.701577 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.702666 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.702667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:14.703718 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:14.703721 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.704560 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.704561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:14.704631 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:14.704635 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:25:14.704680 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 12:25:14.704719 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232}
I20260928 12:25:14.704875 21527 leader_election.cc:302] T 526dbf271b144fa78982f0b984472199 P c5aad6f357ab4db8a937eb0f65bc806a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a4074e801f844c58f76cfc8c9415b71, c5aad6f357ab4db8a937eb0f65bc806a; no voters:
I20260928 12:25:14.705054 21929 raft_consensus.cc:2781] T 526dbf271b144fa78982f0b984472199 P c5aad6f357ab4db8a937eb0f65bc806a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:25:14.705057 21920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:14.701041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:52853 (local address 127.2.56.1:44023)
0928 12:25:14.701072 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:14.701073 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:14.701416 (+ 343us) server_negotiation.cc:407] Connection header received
0928 12:25:14.701436 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:14.701436 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:14.701445 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:14.701484 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:25:14.701665 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.702584 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.703801 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.704278 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.704328 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:14.704336 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:14.704973 (+ 637us) server_negotiation.cc:299] Negotiation successful
0928 12:25:14.705007 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":17}
I20260928 12:25:14.705179 21931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:14.701113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:32823 (local address 127.2.56.2:60073)
0928 12:25:14.701350 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:14.701360 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:14.701405 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:14.701596 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:14.701597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:14.701598 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:25:14.701651 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:14.701653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.702488 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.702489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:14.703645 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:14.703647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.704383 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.704384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:14.704454 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:14.705011 (+ 557us) client_negotiation.cc:769] Sending connection context
0928 12:25:14.705063 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 12:25:14.705103 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":69,"threads_started":1}
I20260928 12:25:14.705117 21929 raft_consensus.cc:482] T 526dbf271b144fa78982f0b984472199 P c5aad6f357ab4db8a937eb0f65bc806a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:25:14.705216 21929 raft_consensus.cc:3037] T 526dbf271b144fa78982f0b984472199 P c5aad6f357ab4db8a937eb0f65bc806a [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:14.705322 21593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526dbf271b144fa78982f0b984472199" candidate_uuid: "6a4074e801f844c58f76cfc8c9415b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5aad6f357ab4db8a937eb0f65bc806a" is_pre_election: true
I20260928 12:25:14.705427 21932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:14.701154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:60073 (local address 127.2.56.3:32823)
0928 12:25:14.701483 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:14.701485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:14.701491 (+ 6us) server_negotiation.cc:407] Connection header received
0928 12:25:14.701508 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:14.701508 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:14.701515 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:14.701564 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:25:14.701722 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.702403 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.703735 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.704280 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.704340 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:14.704351 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:14.705087 (+ 736us) server_negotiation.cc:299] Negotiation successful
0928 12:25:14.705357 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":135,"threads_started":1}
I20260928 12:25:14.705580 21861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526dbf271b144fa78982f0b984472199" candidate_uuid: "6a4074e801f844c58f76cfc8c9415b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce33b920ea03429892c0474c2312783a" is_pre_election: true
I20260928 12:25:14.705641 21861 raft_consensus.cc:2454] T 526dbf271b144fa78982f0b984472199 P ce33b920ea03429892c0474c2312783a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a4074e801f844c58f76cfc8c9415b71 in term 0.
I20260928 12:25:14.705775 21660 leader_election.cc:302] T 526dbf271b144fa78982f0b984472199 P 6a4074e801f844c58f76cfc8c9415b71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a4074e801f844c58f76cfc8c9415b71, ce33b920ea03429892c0474c2312783a; no voters:
I20260928 12:25:14.705886 21927 raft_consensus.cc:2781] T 526dbf271b144fa78982f0b984472199 P 6a4074e801f844c58f76cfc8c9415b71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:25:14.705925 21927 raft_consensus.cc:482] T 526dbf271b144fa78982f0b984472199 P 6a4074e801f844c58f76cfc8c9415b71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:25:14.705945 21927 raft_consensus.cc:3037] T 526dbf271b144fa78982f0b984472199 P 6a4074e801f844c58f76cfc8c9415b71 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:14.706178 21929 raft_consensus.cc:504] T 526dbf271b144fa78982f0b984472199 P c5aad6f357ab4db8a937eb0f65bc806a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5aad6f357ab4db8a937eb0f65bc806a" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44023 } } peers { permanent_uuid: "ce33b920ea03429892c0474c2312783a" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 32823 } } peers { permanent_uuid: "6a4074e801f844c58f76cfc8c9415b71" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 37585 } }
I20260928 12:25:14.706306 21593 raft_consensus.cc:2379] T 526dbf271b144fa78982f0b984472199 P c5aad6f357ab4db8a937eb0f65bc806a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6a4074e801f844c58f76cfc8c9415b71 in current term 1: Already voted for candidate c5aad6f357ab4db8a937eb0f65bc806a in this term.
I20260928 12:25:14.706431 21727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526dbf271b144fa78982f0b984472199" candidate_uuid: "c5aad6f357ab4db8a937eb0f65bc806a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a4074e801f844c58f76cfc8c9415b71"
I20260928 12:25:14.706545 21929 leader_election.cc:288] T 526dbf271b144fa78982f0b984472199 P c5aad6f357ab4db8a937eb0f65bc806a [CANDIDATE]: Term 1 election: Requested vote from peers ce33b920ea03429892c0474c2312783a (127.2.56.3:32823), 6a4074e801f844c58f76cfc8c9415b71 (127.2.56.2:37585)
I20260928 12:25:14.706669 21927 raft_consensus.cc:504] T 526dbf271b144fa78982f0b984472199 P 6a4074e801f844c58f76cfc8c9415b71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5aad6f357ab4db8a937eb0f65bc806a" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44023 } } peers { permanent_uuid: "ce33b920ea03429892c0474c2312783a" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 32823 } } peers { permanent_uuid: "6a4074e801f844c58f76cfc8c9415b71" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 37585 } }
I20260928 12:25:14.706760 21727 raft_consensus.cc:2379] T 526dbf271b144fa78982f0b984472199 P 6a4074e801f844c58f76cfc8c9415b71 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c5aad6f357ab4db8a937eb0f65bc806a in current term 1: Already voted for candidate 6a4074e801f844c58f76cfc8c9415b71 in this term.
I20260928 12:25:14.706775 21927 leader_election.cc:288] T 526dbf271b144fa78982f0b984472199 P 6a4074e801f844c58f76cfc8c9415b71 [CANDIDATE]: Term 1 election: Requested vote from peers c5aad6f357ab4db8a937eb0f65bc806a (127.2.56.1:44023), ce33b920ea03429892c0474c2312783a (127.2.56.3:32823)
I20260928 12:25:14.706837 21861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526dbf271b144fa78982f0b984472199" candidate_uuid: "c5aad6f357ab4db8a937eb0f65bc806a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce33b920ea03429892c0474c2312783a"
I20260928 12:25:14.706900 21861 raft_consensus.cc:3037] T 526dbf271b144fa78982f0b984472199 P ce33b920ea03429892c0474c2312783a [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:14.707016 21860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526dbf271b144fa78982f0b984472199" candidate_uuid: "6a4074e801f844c58f76cfc8c9415b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce33b920ea03429892c0474c2312783a"
I20260928 12:25:14.707031 21593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526dbf271b144fa78982f0b984472199" candidate_uuid: "6a4074e801f844c58f76cfc8c9415b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5aad6f357ab4db8a937eb0f65bc806a"
I20260928 12:25:14.707096 21593 raft_consensus.cc:2379] T 526dbf271b144fa78982f0b984472199 P c5aad6f357ab4db8a937eb0f65bc806a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6a4074e801f844c58f76cfc8c9415b71 in current term 1: Already voted for candidate c5aad6f357ab4db8a937eb0f65bc806a in this term.
I20260928 12:25:14.707231 21660 leader_election.cc:302] T 526dbf271b144fa78982f0b984472199 P 6a4074e801f844c58f76cfc8c9415b71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a4074e801f844c58f76cfc8c9415b71; no voters: c5aad6f357ab4db8a937eb0f65bc806a, ce33b920ea03429892c0474c2312783a
I20260928 12:25:14.707330 21927 raft_consensus.cc:2726] T 526dbf271b144fa78982f0b984472199 P 6a4074e801f844c58f76cfc8c9415b71 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260928 12:25:14.707662 21861 raft_consensus.cc:2454] T 526dbf271b144fa78982f0b984472199 P ce33b920ea03429892c0474c2312783a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5aad6f357ab4db8a937eb0f65bc806a in term 1.
I20260928 12:25:14.707821 21526 leader_election.cc:302] T 526dbf271b144fa78982f0b984472199 P c5aad6f357ab4db8a937eb0f65bc806a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5aad6f357ab4db8a937eb0f65bc806a, ce33b920ea03429892c0474c2312783a; no voters: 6a4074e801f844c58f76cfc8c9415b71
I20260928 12:25:14.707899 21929 raft_consensus.cc:2781] T 526dbf271b144fa78982f0b984472199 P c5aad6f357ab4db8a937eb0f65bc806a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:25:14.708029 21929 raft_consensus.cc:686] T 526dbf271b144fa78982f0b984472199 P c5aad6f357ab4db8a937eb0f65bc806a [term 1 LEADER]: Becoming Leader. State: Replica: c5aad6f357ab4db8a937eb0f65bc806a, State: Running, Role: LEADER
I20260928 12:25:14.708101 21929 consensus_queue.cc:237] T 526dbf271b144fa78982f0b984472199 P c5aad6f357ab4db8a937eb0f65bc806a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5aad6f357ab4db8a937eb0f65bc806a" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44023 } } peers { permanent_uuid: "ce33b920ea03429892c0474c2312783a" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 32823 } } peers { permanent_uuid: "6a4074e801f844c58f76cfc8c9415b71" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 37585 } }
I20260928 12:25:14.708870 21447 catalog_manager.cc:5495] T 526dbf271b144fa78982f0b984472199 P c5aad6f357ab4db8a937eb0f65bc806a reported cstate change: term changed from 0 to 1, leader changed from <none> to c5aad6f357ab4db8a937eb0f65bc806a (127.2.56.1). New cstate: current_term: 1 leader_uuid: "c5aad6f357ab4db8a937eb0f65bc806a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5aad6f357ab4db8a937eb0f65bc806a" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 44023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce33b920ea03429892c0474c2312783a" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 32823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a4074e801f844c58f76cfc8c9415b71" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 37585 } health_report { overall_health: UNKNOWN } } }
I20260928 12:25:14.730942 21920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:14.728543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45248 (local address 127.2.56.1:44023)
0928 12:25:14.728599 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:14.728602 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:14.728609 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:25:14.728645 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:14.728645 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:14.728662 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:14.728718 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:25:14.728828 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.729355 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.730189 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.730422 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.730474 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:14.730627 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:25:14.730729 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:25:14.730784 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:14.730831 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 12:25:14.730849 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 12:25:14.732664 21727 raft_consensus.cc:1261] T 526dbf271b144fa78982f0b984472199 P 6a4074e801f844c58f76cfc8c9415b71 [term 1 FOLLOWER]: Refusing update from remote peer c5aad6f357ab4db8a937eb0f65bc806a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:25:14.732689 21861 raft_consensus.cc:1261] T 526dbf271b144fa78982f0b984472199 P ce33b920ea03429892c0474c2312783a [term 1 FOLLOWER]: Refusing update from remote peer c5aad6f357ab4db8a937eb0f65bc806a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:25:14.732952 21933 consensus_queue.cc:1035] T 526dbf271b144fa78982f0b984472199 P c5aad6f357ab4db8a937eb0f65bc806a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce33b920ea03429892c0474c2312783a" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 32823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:25:14.733243 21933 consensus_queue.cc:1035] T 526dbf271b144fa78982f0b984472199 P c5aad6f357ab4db8a937eb0f65bc806a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a4074e801f844c58f76cfc8c9415b71" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:25:14.735895 21936 mvcc.cc:205] Tried to move back new op lower bound from 7334290697142427648 to 7334290697044824064. Current Snapshot: MvccSnapshot[applied={T|T < 7334290697142427648}]
I20260928 12:25:14.736502 21937 mvcc.cc:205] Tried to move back new op lower bound from 7334290697142427648 to 7334290697044824064. Current Snapshot: MvccSnapshot[applied={T|T < 7334290697142427648}]
I20260928 12:25:14.738265 21448 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:25:14.738358 21448 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:25:14.739053 21448 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c5bea9dcee84d5498e5f2dc90a33671: Sending DeleteTablet for 3 replicas of tablet 526dbf271b144fa78982f0b984472199
I20260928 12:25:14.739388 21707 tablet_service.cc:1514] Processing DeleteTablet for tablet 526dbf271b144fa78982f0b984472199 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:25:14 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:47262
I20260928 12:25:14.739416 21573 tablet_service.cc:1514] Processing DeleteTablet for tablet 526dbf271b144fa78982f0b984472199 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:25:14 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:58652
I20260928 12:25:14.739620 21841 tablet_service.cc:1514] Processing DeleteTablet for tablet 526dbf271b144fa78982f0b984472199 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:25:14 UTC) from {username='kudu', principal='kudu/127.2.56.62@KRBTEST.COM'} at 127.0.0.1:36328
I20260928 12:25:14.739732 21947 tablet_replica.cc:330] T 526dbf271b144fa78982f0b984472199 P c5aad6f357ab4db8a937eb0f65bc806a: stopping tablet replica
I20260928 12:25:14.739807 21948 tablet_replica.cc:330] T 526dbf271b144fa78982f0b984472199 P ce33b920ea03429892c0474c2312783a: stopping tablet replica
I20260928 12:25:14.739821 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 21510
I20260928 12:25:14.739820 21947 raft_consensus.cc:2229] T 526dbf271b144fa78982f0b984472199 P c5aad6f357ab4db8a937eb0f65bc806a [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:25:14.740314 21946 tablet_replica.cc:330] T 526dbf271b144fa78982f0b984472199 P 6a4074e801f844c58f76cfc8c9415b71: stopping tablet replica
I20260928 12:25:14.740394 21946 raft_consensus.cc:2229] T 526dbf271b144fa78982f0b984472199 P 6a4074e801f844c58f76cfc8c9415b71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:25:14.740475 21946 raft_consensus.cc:2258] T 526dbf271b144fa78982f0b984472199 P 6a4074e801f844c58f76cfc8c9415b71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:25:14.740870 21946 ts_tablet_manager.cc:1905] T 526dbf271b144fa78982f0b984472199 P 6a4074e801f844c58f76cfc8c9415b71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:25:14.739888 21948 raft_consensus.cc:2229] T 526dbf271b144fa78982f0b984472199 P ce33b920ea03429892c0474c2312783a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:25:14.741323 21948 raft_consensus.cc:2258] T 526dbf271b144fa78982f0b984472199 P ce33b920ea03429892c0474c2312783a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:25:14.741675 21948 ts_tablet_manager.cc:1905] T 526dbf271b144fa78982f0b984472199 P ce33b920ea03429892c0474c2312783a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:25:14.742564 21946 ts_tablet_manager.cc:1918] T 526dbf271b144fa78982f0b984472199 P 6a4074e801f844c58f76cfc8c9415b71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:25:14.742635 21946 log.cc:1196] T 526dbf271b144fa78982f0b984472199 P 6a4074e801f844c58f76cfc8c9415b71: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-1/wal/wals/526dbf271b144fa78982f0b984472199
I20260928 12:25:14.742880 21946 ts_tablet_manager.cc:1939] T 526dbf271b144fa78982f0b984472199 P 6a4074e801f844c58f76cfc8c9415b71: Deleting consensus metadata
I20260928 12:25:14.743268 21433 catalog_manager.cc:4841] TS 6a4074e801f844c58f76cfc8c9415b71 (127.2.56.2:37585): tablet 526dbf271b144fa78982f0b984472199 (table test-table [id=c795dfda228942bdac756dc55dbcaaa0]) successfully deleted
I20260928 12:25:14.743505 21948 ts_tablet_manager.cc:1918] T 526dbf271b144fa78982f0b984472199 P ce33b920ea03429892c0474c2312783a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:25:14.743575 21948 log.cc:1196] T 526dbf271b144fa78982f0b984472199 P ce33b920ea03429892c0474c2312783a: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598072638673-2272-0/minicluster-data/ts-2/wal/wals/526dbf271b144fa78982f0b984472199
I20260928 12:25:14.743796 21948 ts_tablet_manager.cc:1939] T 526dbf271b144fa78982f0b984472199 P ce33b920ea03429892c0474c2312783a: Deleting consensus metadata
I20260928 12:25:14.744154 21432 catalog_manager.cc:4841] TS ce33b920ea03429892c0474c2312783a (127.2.56.3:32823): tablet 526dbf271b144fa78982f0b984472199 (table test-table [id=c795dfda228942bdac756dc55dbcaaa0]) successfully deleted
W20260928 12:25:14.745463 21432 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.56.1:44023 (error 108)
I20260928 12:25:14.745636 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 21644
I20260928 12:25:14.746258 21916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:14.745755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:44023 (local address 127.0.0.1:45256)
0928 12:25:14.746186 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:14.746223 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.1:44023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407}
W20260928 12:25:14.746416 21432 catalog_manager.cc:4584] TS c5aad6f357ab4db8a937eb0f65bc806a (127.2.56.1:44023): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 526dbf271b144fa78982f0b984472199: Network error: Client connection negotiation failed: client connection to 127.2.56.1:44023: connect: Connection refused (error 111)
I20260928 12:25:14.750490 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 21778
I20260928 12:25:14.754279 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 21416
2026-09-28T12:25:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1854 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T12:25:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:25:14Z Disabled control of system clock
I20260928 12:25:14.780016 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:37573
--webserver_interface=127.2.56.62
--webserver_port=0
--builtin_ntp_servers=127.2.56.20:42779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:37573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 12:25:14.857369 21953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:14.857573 21953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:14.857594 21953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:14.858673 21953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:25:14.858706 21953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:14.858717 21953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:14.858728 21953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:25:14.858738 21953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:25:14.859992 21953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:42779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:37573
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:37573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:14.860180 21953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:14.860391 21953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:14.864687 21959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:14.864800 21958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:14.864694 21961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:14.864884 21953 server_base.cc:1034] running on GCE node
I20260928 12:25:14.865113 21953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:14.865406 21953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:14.866570 21953 hybrid_clock.cc:648] HybridClock initialized: now 1790598314866551 us; error 28 us; skew 500 ppm
I20260928 12:25:14.867774 21953 webserver.cc:460] Webserver started at http://127.2.56.62:38971/ using document root <none> and password file <none>
I20260928 12:25:14.868059 21953 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:14.868139 21953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:14.868268 21953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:14.869386 21953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "15858f7da4af4604a8b69562bd9109cb"
format_stamp: "Formatted at 2026-09-28 12:25:14 on dist-test-slave-920p"
server_key: "c9eaaa8b17609d6221b5759f23f83a30"
server_key_iv: "a5c5bd564ae622aed74fc0b29989cacb"
server_key_version: "encryptionkey@0"
I20260928 12:25:14.869710 21953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "15858f7da4af4604a8b69562bd9109cb"
format_stamp: "Formatted at 2026-09-28 12:25:14 on dist-test-slave-920p"
server_key: "c9eaaa8b17609d6221b5759f23f83a30"
server_key_iv: "a5c5bd564ae622aed74fc0b29989cacb"
server_key_version: "encryptionkey@0"
I20260928 12:25:14.870831 21953 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:25:14.871589 21967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:14.871753 21953 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:25:14.871834 21953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "15858f7da4af4604a8b69562bd9109cb"
format_stamp: "Formatted at 2026-09-28 12:25:14 on dist-test-slave-920p"
server_key: "c9eaaa8b17609d6221b5759f23f83a30"
server_key_iv: "a5c5bd564ae622aed74fc0b29989cacb"
server_key_version: "encryptionkey@0"
I20260928 12:25:14.871898 21953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:14.888659 21953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:14.888931 21953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:14.889058 21953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:14.892455 21953 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:37573
I20260928 12:25:14.892489 22019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:37573 every 8 connection(s)
I20260928 12:25:14.892773 21953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:25:14.893690 22020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:14.893778 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 21953
I20260928 12:25:14.893877 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:25:14.894068 2272 external_mini_cluster.cc:1428] Setting key e3c080a13d4ab7480b9f5fb509d2101a
I20260928 12:25:14.896826 22020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15858f7da4af4604a8b69562bd9109cb: Bootstrap starting.
I20260928 12:25:14.897423 22020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15858f7da4af4604a8b69562bd9109cb: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:14.897678 22020 log.cc:824] T 00000000000000000000000000000000 P 15858f7da4af4604a8b69562bd9109cb: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:14.898337 22020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15858f7da4af4604a8b69562bd9109cb: No bootstrap required, opened a new log
I20260928 12:25:14.899585 22023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:14.894519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49462 (local address 127.2.56.62:37573)
0928 12:25:14.894732 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:14.894741 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:14.894774 (+ 33us) server_negotiation.cc:407] Connection header received
0928 12:25:14.895225 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:14.895233 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:14.895308 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:14.895442 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:14.895618 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.896892 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.897713 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:14.898233 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:14.898348 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:14.898930 (+ 582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:14.898934 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:14.898934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:14.899120 (+ 186us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:14.899169 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:14.899183 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:14.899245 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:14.899311 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 12:25:14.899385 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
I20260928 12:25:14.900398 22020 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15858f7da4af4604a8b69562bd9109cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15858f7da4af4604a8b69562bd9109cb" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 37573 } }
I20260928 12:25:14.900523 22020 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15858f7da4af4604a8b69562bd9109cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:14.900543 22020 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15858f7da4af4604a8b69562bd9109cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15858f7da4af4604a8b69562bd9109cb, State: Initialized, Role: FOLLOWER
I20260928 12:25:14.900652 22020 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15858f7da4af4604a8b69562bd9109cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15858f7da4af4604a8b69562bd9109cb" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 37573 } }
I20260928 12:25:14.900727 22020 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15858f7da4af4604a8b69562bd9109cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:25:14.900758 22020 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15858f7da4af4604a8b69562bd9109cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:25:14.900803 22020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15858f7da4af4604a8b69562bd9109cb [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:14.901342 22020 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15858f7da4af4604a8b69562bd9109cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15858f7da4af4604a8b69562bd9109cb" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 37573 } }
I20260928 12:25:14.901429 22020 leader_election.cc:302] T 00000000000000000000000000000000 P 15858f7da4af4604a8b69562bd9109cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15858f7da4af4604a8b69562bd9109cb; no voters:
I20260928 12:25:14.901576 22020 leader_election.cc:288] T 00000000000000000000000000000000 P 15858f7da4af4604a8b69562bd9109cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:25:14.901619 22025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15858f7da4af4604a8b69562bd9109cb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:25:14.901751 22025 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15858f7da4af4604a8b69562bd9109cb [term 1 LEADER]: Becoming Leader. State: Replica: 15858f7da4af4604a8b69562bd9109cb, State: Running, Role: LEADER
I20260928 12:25:14.901799 22020 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15858f7da4af4604a8b69562bd9109cb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:25:14.901836 22025 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15858f7da4af4604a8b69562bd9109cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15858f7da4af4604a8b69562bd9109cb" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 37573 } }
I20260928 12:25:14.902209 22026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15858f7da4af4604a8b69562bd9109cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15858f7da4af4604a8b69562bd9109cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15858f7da4af4604a8b69562bd9109cb" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 37573 } } }
I20260928 12:25:14.902288 22026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15858f7da4af4604a8b69562bd9109cb [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:14.902464 22027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15858f7da4af4604a8b69562bd9109cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15858f7da4af4604a8b69562bd9109cb. Latest consensus state: current_term: 1 leader_uuid: "15858f7da4af4604a8b69562bd9109cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15858f7da4af4604a8b69562bd9109cb" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 37573 } } }
I20260928 12:25:14.902518 22027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15858f7da4af4604a8b69562bd9109cb [sys.catalog]: This master's current role is: LEADER
W20260928 12:25:14.903335 22041 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 15858f7da4af4604a8b69562bd9109cb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:25:14.903376 22041 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:25:14.903427 22034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:25:14.903581 22034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:25:14.904838 22034 catalog_manager.cc:1347] Generated new cluster ID: e21339f699aa44949ca92234b77eb501
I20260928 12:25:14.904886 22034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:25:14.925222 22034 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:25:14.925748 22034 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:25:14.929970 22034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15858f7da4af4604a8b69562bd9109cb: Generated new TSK 0
I20260928 12:25:14.930111 22034 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:25:14.933837 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.1:0
--local_ip_for_outbound_sockets=127.2.56.1
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:37573
--builtin_ntp_servers=127.2.56.20:42779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 12:25:15.014025 22044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:15.014201 22044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:15.014221 22044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:15.015321 22044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:15.015357 22044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:15.015381 22044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.1
I20260928 12:25:15.016680 22044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:42779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.56.1
--webserver_port=0
--tserver_master_addrs=127.2.56.62:37573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.1
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:15.016891 22044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:15.017153 22044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:15.022596 22049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:15.022691 22052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:15.022626 22050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:15.022897 22044 server_base.cc:1034] running on GCE node
I20260928 12:25:15.023238 22044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:15.023506 22044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:15.024655 22044 hybrid_clock.cc:648] HybridClock initialized: now 1790598315024641 us; error 29 us; skew 500 ppm
I20260928 12:25:15.025807 22044 webserver.cc:460] Webserver started at http://127.2.56.1:34001/ using document root <none> and password file <none>
I20260928 12:25:15.026039 22044 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:15.026113 22044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:15.026243 22044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:15.027235 22044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-0/data/instance:
uuid: "88ef4913f075441ab858aaa59f83f157"
format_stamp: "Formatted at 2026-09-28 12:25:15 on dist-test-slave-920p"
server_key: "2aa93b62f32caeeae59138dca0d33e75"
server_key_iv: "9072b34d9bdaaf829df7dc96287fe0a5"
server_key_version: "encryptionkey@0"
I20260928 12:25:15.027536 22044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance:
uuid: "88ef4913f075441ab858aaa59f83f157"
format_stamp: "Formatted at 2026-09-28 12:25:15 on dist-test-slave-920p"
server_key: "2aa93b62f32caeeae59138dca0d33e75"
server_key_iv: "9072b34d9bdaaf829df7dc96287fe0a5"
server_key_version: "encryptionkey@0"
I20260928 12:25:15.028692 22044 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:25:15.029361 22058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:15.029493 22044 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:15.029569 22044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-0/wal
uuid: "88ef4913f075441ab858aaa59f83f157"
format_stamp: "Formatted at 2026-09-28 12:25:15 on dist-test-slave-920p"
server_key: "2aa93b62f32caeeae59138dca0d33e75"
server_key_iv: "9072b34d9bdaaf829df7dc96287fe0a5"
server_key_version: "encryptionkey@0"
I20260928 12:25:15.029635 22044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:15.049851 22044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:15.050123 22044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:15.050242 22044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:15.050442 22044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:25:15.050737 22044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:25:15.050769 22044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:15.050797 22044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:25:15.050820 22044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:15.055994 22044 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.1:40769
I20260928 12:25:15.056053 22171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.1:40769 every 8 connection(s)
I20260928 12:25:15.056324 22044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-0/data/info.pb
I20260928 12:25:15.058152 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 22044
I20260928 12:25:15.058260 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-0/wal/instance
I20260928 12:25:15.058444 2272 external_mini_cluster.cc:1428] Setting key 00831148d90684c0cfbb12f68af9145f
I20260928 12:25:15.059324 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.2:0
--local_ip_for_outbound_sockets=127.2.56.2
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:37573
--builtin_ntp_servers=127.2.56.20:42779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 12:25:15.061860 22023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:15.056947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:34023 (local address 127.2.56.62:37573)
0928 12:25:15.057020 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:15.057022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:15.057236 (+ 214us) server_negotiation.cc:407] Connection header received
0928 12:25:15.057707 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:15.057708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:15.057721 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:15.057772 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:15.058520 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:15.059226 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:15.060530 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:15.060827 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:15.060890 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:15.061620 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:15.061621 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:15.061621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:15.061626 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:15.061642 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:15.061646 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:15.061703 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:15.061750 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 12:25:15.061783 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 12:25:15.061997 22174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:15.056940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:37573 (local address 127.2.56.1:34023)
0928 12:25:15.057132 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:15.057163 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:15.057630 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:15.057819 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:15.057822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:15.057829 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:15.058431 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:15.058437 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:15.059312 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:15.059313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:15.060436 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:15.060439 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:15.060906 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:15.060907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:15.061014 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:15.061512 (+ 498us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:25:15.061512 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:15.061539 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:25:15.061561 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:15.061705 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:15.061707 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:15.061709 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:25:15.061750 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 12:25:15.061821 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":96,"threads_started":1}
I20260928 12:25:15.062397 22172 heartbeater.cc:347] Connected to a master server at 127.2.56.62:37573
I20260928 12:25:15.062502 22172 heartbeater.cc:464] Registering TS with master...
I20260928 12:25:15.062683 22172 heartbeater.cc:511] Master 127.2.56.62:37573 requested a full tablet report, sending...
I20260928 12:25:15.063156 21984 ts_manager.cc:194] Registered new tserver with Master: 88ef4913f075441ab858aaa59f83f157 (127.2.56.1:40769)
I20260928 12:25:15.063736 21984 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.56.1:34023
W20260928 12:25:15.137997 22175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:15.138204 22175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:15.138233 22175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:15.139340 22175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:15.139379 22175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:15.139411 22175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.2
I20260928 12:25:15.140712 22175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:42779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.56.2
--webserver_port=0
--tserver_master_addrs=127.2.56.62:37573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.2
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:15.140933 22175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:15.141199 22175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:15.145783 22183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:15.145793 22180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:15.145794 22181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:15.146013 22175 server_base.cc:1034] running on GCE node
I20260928 12:25:15.146198 22175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:15.146418 22175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:15.147557 22175 hybrid_clock.cc:648] HybridClock initialized: now 1790598315147545 us; error 26 us; skew 500 ppm
I20260928 12:25:15.148674 22175 webserver.cc:460] Webserver started at http://127.2.56.2:43473/ using document root <none> and password file <none>
I20260928 12:25:15.148870 22175 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:15.148911 22175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:15.149029 22175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:15.150058 22175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-1/data/instance:
uuid: "404b1ee2892c472d90cd200ff9d136a0"
format_stamp: "Formatted at 2026-09-28 12:25:15 on dist-test-slave-920p"
server_key: "6888d022cbd45b524f10d84830bb501e"
server_key_iv: "b7128d34e8e4976fd1591c0682f5dd02"
server_key_version: "encryptionkey@0"
I20260928 12:25:15.150363 22175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance:
uuid: "404b1ee2892c472d90cd200ff9d136a0"
format_stamp: "Formatted at 2026-09-28 12:25:15 on dist-test-slave-920p"
server_key: "6888d022cbd45b524f10d84830bb501e"
server_key_iv: "b7128d34e8e4976fd1591c0682f5dd02"
server_key_version: "encryptionkey@0"
I20260928 12:25:15.151469 22175 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 12:25:15.152060 22189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:15.152184 22175 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:25:15.152262 22175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-1/wal
uuid: "404b1ee2892c472d90cd200ff9d136a0"
format_stamp: "Formatted at 2026-09-28 12:25:15 on dist-test-slave-920p"
server_key: "6888d022cbd45b524f10d84830bb501e"
server_key_iv: "b7128d34e8e4976fd1591c0682f5dd02"
server_key_version: "encryptionkey@0"
I20260928 12:25:15.152328 22175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:15.161887 22175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:15.162133 22175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:15.162258 22175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:15.162451 22175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:25:15.162737 22175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:25:15.162768 22175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:15.162794 22175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:25:15.162818 22175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:15.168044 22175 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.2:42163
I20260928 12:25:15.168081 22302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.2:42163 every 8 connection(s)
I20260928 12:25:15.168357 22175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-1/data/info.pb
I20260928 12:25:15.173766 22023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:15.168985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.2:34481 (local address 127.2.56.62:37573)
0928 12:25:15.169034 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:15.169036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:15.169242 (+ 206us) server_negotiation.cc:407] Connection header received
0928 12:25:15.169731 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:15.169733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:15.169742 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:15.169795 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:15.170396 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:15.170874 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:15.171840 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:15.172054 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:15.172111 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:15.173246 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:15.173249 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:15.173249 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:15.173258 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:15.173326 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:15.173335 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:15.173457 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:15.173640 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 12:25:15.173684 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 12:25:15.173864 22305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:15.168950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:37573 (local address 127.2.56.2:34481)
0928 12:25:15.169133 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:15.169170 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:15.169653 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:15.169856 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:15.169859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:15.169867 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:15.170338 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:15.170342 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:15.170934 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:15.170934 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:15.171714 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:15.171717 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:15.172166 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:15.172167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:15.172289 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:15.172966 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:25:15.172967 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:15.173002 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:25:15.173025 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:15.173495 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:15.173502 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:15.173505 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:25:15.173590 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 12:25:15.173673 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260928 12:25:15.174211 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 22175
I20260928 12:25:15.174315 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-1/wal/instance
I20260928 12:25:15.174543 2272 external_mini_cluster.cc:1428] Setting key 42a2fa08e1fe7178653af2621a917a34
I20260928 12:25:15.174544 22303 heartbeater.cc:347] Connected to a master server at 127.2.56.62:37573
I20260928 12:25:15.174654 22303 heartbeater.cc:464] Registering TS with master...
I20260928 12:25:15.174860 22303 heartbeater.cc:511] Master 127.2.56.62:37573 requested a full tablet report, sending...
I20260928 12:25:15.175426 21983 ts_manager.cc:194] Registered new tserver with Master: 404b1ee2892c472d90cd200ff9d136a0 (127.2.56.2:42163)
I20260928 12:25:15.175880 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.56.3:0
--local_ip_for_outbound_sockets=127.2.56.3
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:37573
--builtin_ntp_servers=127.2.56.20:42779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 12:25:15.176021 21983 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.56.2:34481
W20260928 12:25:15.253863 22306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:15.254046 22306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:15.254065 22306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:15.255196 22306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:15.255231 22306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:15.255254 22306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.56.3
I20260928 12:25:15.256522 22306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:42779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.56.3
--webserver_port=0
--tserver_master_addrs=127.2.56.62:37573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.56.3
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:15.256690 22306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:15.256908 22306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:15.261258 22311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:15.261389 22314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:15.261479 22312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:15.261487 22306 server_base.cc:1034] running on GCE node
I20260928 12:25:15.261720 22306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:15.261920 22306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:15.263063 22306 hybrid_clock.cc:648] HybridClock initialized: now 1790598315263052 us; error 27 us; skew 500 ppm
I20260928 12:25:15.264169 22306 webserver.cc:460] Webserver started at http://127.2.56.3:41605/ using document root <none> and password file <none>
I20260928 12:25:15.264364 22306 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:15.264410 22306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:15.264523 22306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:15.265544 22306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-2/data/instance:
uuid: "42ca3747fe9c4b2bbe4bbf4890011742"
format_stamp: "Formatted at 2026-09-28 12:25:15 on dist-test-slave-920p"
server_key: "22ed834c3928e420c67fe40692c42df6"
server_key_iv: "2c33407892290c20d0ec4d8e038bdcfa"
server_key_version: "encryptionkey@0"
I20260928 12:25:15.265851 22306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance:
uuid: "42ca3747fe9c4b2bbe4bbf4890011742"
format_stamp: "Formatted at 2026-09-28 12:25:15 on dist-test-slave-920p"
server_key: "22ed834c3928e420c67fe40692c42df6"
server_key_iv: "2c33407892290c20d0ec4d8e038bdcfa"
server_key_version: "encryptionkey@0"
I20260928 12:25:15.266973 22306 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:25:15.267566 22320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:15.267715 22306 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:25:15.267793 22306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-2/wal
uuid: "42ca3747fe9c4b2bbe4bbf4890011742"
format_stamp: "Formatted at 2026-09-28 12:25:15 on dist-test-slave-920p"
server_key: "22ed834c3928e420c67fe40692c42df6"
server_key_iv: "2c33407892290c20d0ec4d8e038bdcfa"
server_key_version: "encryptionkey@0"
I20260928 12:25:15.267848 22306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:15.283731 22306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:15.283986 22306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:15.284099 22306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:15.284283 22306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:25:15.284581 22306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:25:15.284621 22306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:15.284648 22306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:25:15.284672 22306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:15.289965 22306 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.3:33153
I20260928 12:25:15.290016 22433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.3:33153 every 8 connection(s)
I20260928 12:25:15.290287 22306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-2/data/info.pb
I20260928 12:25:15.291075 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 22306
I20260928 12:25:15.291175 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-2/wal/instance
I20260928 12:25:15.291354 2272 external_mini_cluster.cc:1428] Setting key 08c7a9661302ce0aec55ce2cb8ee07dc
I20260928 12:25:15.295066 22023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:15.290918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.3:50973 (local address 127.2.56.62:37573)
0928 12:25:15.290985 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:15.290987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:15.291192 (+ 205us) server_negotiation.cc:407] Connection header received
0928 12:25:15.291767 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:15.291768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:15.291780 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:15.291834 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:15.292569 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:15.293062 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:15.293984 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:15.294178 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:15.294225 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:15.294855 (+ 630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:15.294856 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:15.294856 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:15.294860 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:15.294874 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:15.294878 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:15.294913 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:15.294962 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 12:25:15.294982 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 12:25:15.295244 22436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:15.290860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:37573 (local address 127.2.56.3:50973)
0928 12:25:15.291094 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:15.291124 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:15.291665 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:15.291912 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:15.291917 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:15.291927 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:15.292492 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:15.292497 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:15.293125 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:15.293126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:15.293923 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:15.293925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:15.294232 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:15.294233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:15.294328 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:15.294742 (+ 414us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:25:15.294742 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:15.294767 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:25:15.294787 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:15.294915 (+ 128us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:15.294917 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:15.294919 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:25:15.294957 (+ 38us) client_negotiation.cc:240] Negotiation successful
0928 12:25:15.295050 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":76,"threads_started":1}
I20260928 12:25:15.295608 22434 heartbeater.cc:347] Connected to a master server at 127.2.56.62:37573
I20260928 12:25:15.295689 22434 heartbeater.cc:464] Registering TS with master...
I20260928 12:25:15.295820 22434 heartbeater.cc:511] Master 127.2.56.62:37573 requested a full tablet report, sending...
I20260928 12:25:15.296120 21983 ts_manager.cc:194] Registered new tserver with Master: 42ca3747fe9c4b2bbe4bbf4890011742 (127.2.56.3:33153)
I20260928 12:25:15.296471 21983 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.56.3:50973
I20260928 12:25:15.302964 2272 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:25:15.306636 22023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:15.304226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49466 (local address 127.2.56.62:37573)
0928 12:25:15.304277 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:15.304279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:15.304369 (+ 90us) server_negotiation.cc:407] Connection header received
0928 12:25:15.304407 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:15.304407 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:15.304415 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:15.304457 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:15.304584 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:15.305020 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:15.305739 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:15.305912 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:15.305957 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:15.306460 (+ 503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:15.306461 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:15.306461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:15.306465 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:15.306477 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:15.306481 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:15.306515 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:15.306546 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 12:25:15.306564 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 12:25:15.307994 21983 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49466:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:25:15.309494 21983 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:25:15.315461 22444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:15.311419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33390 (local address 127.2.56.1:40769)
0928 12:25:15.311876 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:15.311880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:15.311890 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:25:15.311928 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:15.311930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:15.311944 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:15.312017 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:25:15.312202 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:15.313316 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:15.314290 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:15.315046 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:15.315125 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:15.315215 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:15.315347 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 12:25:15.315383 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":79,"threads_started":1}
I20260928 12:25:15.315497 22442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:15.311304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:33153 (local address 127.0.0.1:55450)
0928 12:25:15.311407 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:15.311446 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:15.311508 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:15.311780 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:15.311782 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:15.311784 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:25:15.311844 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:15.311846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:15.313204 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:15.313205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:15.314280 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:15.314281 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:15.315145 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:15.315147 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:15.315237 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:15.315251 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 12:25:15.315329 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 12:25:15.315405 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":43,"thread_start_us":57,"threads_started":1}
I20260928 12:25:15.315497 22443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:15.311316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:40769 (local address 127.0.0.1:33390)
0928 12:25:15.311607 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:15.311616 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:15.311670 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:15.312122 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:15.312122 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:15.312124 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:25:15.312148 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:15.312149 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:15.313380 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:15.313381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:15.314234 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:15.314236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:15.315147 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:15.315147 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:15.315237 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:15.315251 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 12:25:15.315329 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 12:25:15.315405 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":2,"thread_start_us":81,"threads_started":1}
I20260928 12:25:15.315867 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:15.311453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55450 (local address 127.2.56.3:33153)
0928 12:25:15.311624 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:15.311633 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:15.311643 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:25:15.311679 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:15.311681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:15.311696 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:15.311775 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:25:15.311911 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:15.313130 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:15.314350 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:15.315042 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:15.315125 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:15.315215 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:15.315740 (+ 525us) server_negotiation.cc:299] Negotiation successful
0928 12:25:15.315780 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":85,"threads_started":1}
I20260928 12:25:15.316514 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:15.311490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:42163 (local address 127.0.0.1:52322)
0928 12:25:15.311686 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:15.311695 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:15.311730 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:15.312014 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:15.312015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:15.312016 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:25:15.312054 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:15.312056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:15.313032 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:15.313033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:15.314278 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:15.314281 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:15.316297 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:15.316298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:15.316361 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:15.316366 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:25:15.316410 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 12:25:15.316449 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":56,"threads_started":1}
I20260928 12:25:15.316460 22106 tablet_service.cc:1467] Processing CreateTablet for tablet 7b19ddec564742ef97c261abb16bb4af (DEFAULT_TABLE table=test-table [id=aae07935fe9441a9b9f2bfce8095c149]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:25:15.316485 22368 tablet_service.cc:1467] Processing CreateTablet for tablet 7b19ddec564742ef97c261abb16bb4af (DEFAULT_TABLE table=test-table [id=aae07935fe9441a9b9f2bfce8095c149]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:25:15.316788 22447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:15.311681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52322 (local address 127.2.56.2:42163)
0928 12:25:15.311865 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:15.311869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:15.311877 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:25:15.311919 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:15.311921 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:15.311935 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:15.312012 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:25:15.312221 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:15.312951 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:15.315251 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:15.316116 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:15.316180 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:15.316263 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:15.316675 (+ 412us) server_negotiation.cc:299] Negotiation successful
0928 12:25:15.316713 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":87,"threads_started":1}
I20260928 12:25:15.316851 22368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b19ddec564742ef97c261abb16bb4af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:15.316879 22106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b19ddec564742ef97c261abb16bb4af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:15.317399 22237 tablet_service.cc:1467] Processing CreateTablet for tablet 7b19ddec564742ef97c261abb16bb4af (DEFAULT_TABLE table=test-table [id=aae07935fe9441a9b9f2bfce8095c149]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:25:15.317698 22237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b19ddec564742ef97c261abb16bb4af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:15.319303 22448 tablet_bootstrap.cc:492] T 7b19ddec564742ef97c261abb16bb4af P 88ef4913f075441ab858aaa59f83f157: Bootstrap starting.
I20260928 12:25:15.319788 22448 tablet_bootstrap.cc:654] T 7b19ddec564742ef97c261abb16bb4af P 88ef4913f075441ab858aaa59f83f157: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:15.320003 22449 tablet_bootstrap.cc:492] T 7b19ddec564742ef97c261abb16bb4af P 42ca3747fe9c4b2bbe4bbf4890011742: Bootstrap starting.
I20260928 12:25:15.320060 22448 log.cc:824] T 7b19ddec564742ef97c261abb16bb4af P 88ef4913f075441ab858aaa59f83f157: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:15.320528 22449 tablet_bootstrap.cc:654] T 7b19ddec564742ef97c261abb16bb4af P 42ca3747fe9c4b2bbe4bbf4890011742: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:15.320704 22451 tablet_bootstrap.cc:492] T 7b19ddec564742ef97c261abb16bb4af P 404b1ee2892c472d90cd200ff9d136a0: Bootstrap starting.
I20260928 12:25:15.320830 22449 log.cc:824] T 7b19ddec564742ef97c261abb16bb4af P 42ca3747fe9c4b2bbe4bbf4890011742: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:15.320974 22448 tablet_bootstrap.cc:492] T 7b19ddec564742ef97c261abb16bb4af P 88ef4913f075441ab858aaa59f83f157: No bootstrap required, opened a new log
I20260928 12:25:15.321034 22448 ts_tablet_manager.cc:1397] T 7b19ddec564742ef97c261abb16bb4af P 88ef4913f075441ab858aaa59f83f157: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:25:15.321314 22451 tablet_bootstrap.cc:654] T 7b19ddec564742ef97c261abb16bb4af P 404b1ee2892c472d90cd200ff9d136a0: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:15.321568 22451 log.cc:824] T 7b19ddec564742ef97c261abb16bb4af P 404b1ee2892c472d90cd200ff9d136a0: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:15.321794 22449 tablet_bootstrap.cc:492] T 7b19ddec564742ef97c261abb16bb4af P 42ca3747fe9c4b2bbe4bbf4890011742: No bootstrap required, opened a new log
I20260928 12:25:15.321858 22449 ts_tablet_manager.cc:1397] T 7b19ddec564742ef97c261abb16bb4af P 42ca3747fe9c4b2bbe4bbf4890011742: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:25:15.322104 22451 tablet_bootstrap.cc:492] T 7b19ddec564742ef97c261abb16bb4af P 404b1ee2892c472d90cd200ff9d136a0: No bootstrap required, opened a new log
I20260928 12:25:15.322157 22451 ts_tablet_manager.cc:1397] T 7b19ddec564742ef97c261abb16bb4af P 404b1ee2892c472d90cd200ff9d136a0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 12:25:15.323450 22448 raft_consensus.cc:348] T 7b19ddec564742ef97c261abb16bb4af P 88ef4913f075441ab858aaa59f83f157 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42ca3747fe9c4b2bbe4bbf4890011742" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 33153 } } peers { permanent_uuid: "88ef4913f075441ab858aaa59f83f157" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 40769 } } peers { permanent_uuid: "404b1ee2892c472d90cd200ff9d136a0" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 42163 } }
I20260928 12:25:15.323616 22448 raft_consensus.cc:374] T 7b19ddec564742ef97c261abb16bb4af P 88ef4913f075441ab858aaa59f83f157 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:15.323652 22448 raft_consensus.cc:729] T 7b19ddec564742ef97c261abb16bb4af P 88ef4913f075441ab858aaa59f83f157 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88ef4913f075441ab858aaa59f83f157, State: Initialized, Role: FOLLOWER
I20260928 12:25:15.323757 22448 consensus_queue.cc:260] T 7b19ddec564742ef97c261abb16bb4af P 88ef4913f075441ab858aaa59f83f157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42ca3747fe9c4b2bbe4bbf4890011742" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 33153 } } peers { permanent_uuid: "88ef4913f075441ab858aaa59f83f157" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 40769 } } peers { permanent_uuid: "404b1ee2892c472d90cd200ff9d136a0" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 42163 } }
I20260928 12:25:15.324002 22448 ts_tablet_manager.cc:1428] T 7b19ddec564742ef97c261abb16bb4af P 88ef4913f075441ab858aaa59f83f157: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:25:15.323915 22451 raft_consensus.cc:348] T 7b19ddec564742ef97c261abb16bb4af P 404b1ee2892c472d90cd200ff9d136a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42ca3747fe9c4b2bbe4bbf4890011742" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 33153 } } peers { permanent_uuid: "88ef4913f075441ab858aaa59f83f157" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 40769 } } peers { permanent_uuid: "404b1ee2892c472d90cd200ff9d136a0" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 42163 } }
I20260928 12:25:15.324038 22172 heartbeater.cc:503] Master 127.2.56.62:37573 was elected leader, sending a full tablet report...
I20260928 12:25:15.324054 22451 raft_consensus.cc:374] T 7b19ddec564742ef97c261abb16bb4af P 404b1ee2892c472d90cd200ff9d136a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:15.324083 22451 raft_consensus.cc:729] T 7b19ddec564742ef97c261abb16bb4af P 404b1ee2892c472d90cd200ff9d136a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 404b1ee2892c472d90cd200ff9d136a0, State: Initialized, Role: FOLLOWER
I20260928 12:25:15.324198 22451 consensus_queue.cc:260] T 7b19ddec564742ef97c261abb16bb4af P 404b1ee2892c472d90cd200ff9d136a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42ca3747fe9c4b2bbe4bbf4890011742" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 33153 } } peers { permanent_uuid: "88ef4913f075441ab858aaa59f83f157" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 40769 } } peers { permanent_uuid: "404b1ee2892c472d90cd200ff9d136a0" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 42163 } }
I20260928 12:25:15.324175 22449 raft_consensus.cc:348] T 7b19ddec564742ef97c261abb16bb4af P 42ca3747fe9c4b2bbe4bbf4890011742 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42ca3747fe9c4b2bbe4bbf4890011742" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 33153 } } peers { permanent_uuid: "88ef4913f075441ab858aaa59f83f157" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 40769 } } peers { permanent_uuid: "404b1ee2892c472d90cd200ff9d136a0" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 42163 } }
I20260928 12:25:15.324285 22449 raft_consensus.cc:374] T 7b19ddec564742ef97c261abb16bb4af P 42ca3747fe9c4b2bbe4bbf4890011742 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:15.324323 22449 raft_consensus.cc:729] T 7b19ddec564742ef97c261abb16bb4af P 42ca3747fe9c4b2bbe4bbf4890011742 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42ca3747fe9c4b2bbe4bbf4890011742, State: Initialized, Role: FOLLOWER
I20260928 12:25:15.324424 22449 consensus_queue.cc:260] T 7b19ddec564742ef97c261abb16bb4af P 42ca3747fe9c4b2bbe4bbf4890011742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42ca3747fe9c4b2bbe4bbf4890011742" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 33153 } } peers { permanent_uuid: "88ef4913f075441ab858aaa59f83f157" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 40769 } } peers { permanent_uuid: "404b1ee2892c472d90cd200ff9d136a0" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 42163 } }
I20260928 12:25:15.324579 22451 ts_tablet_manager.cc:1428] T 7b19ddec564742ef97c261abb16bb4af P 404b1ee2892c472d90cd200ff9d136a0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 12:25:15.324652 22449 ts_tablet_manager.cc:1428] T 7b19ddec564742ef97c261abb16bb4af P 42ca3747fe9c4b2bbe4bbf4890011742: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:25:15.324767 22434 heartbeater.cc:503] Master 127.2.56.62:37573 was elected leader, sending a full tablet report...
I20260928 12:25:15.324982 22303 heartbeater.cc:503] Master 127.2.56.62:37573 was elected leader, sending a full tablet report...
W20260928 12:25:15.418848 22304 tablet.cc:2369] T 7b19ddec564742ef97c261abb16bb4af P 404b1ee2892c472d90cd200ff9d136a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:25:15.474455 22454 raft_consensus.cc:482] T 7b19ddec564742ef97c261abb16bb4af P 88ef4913f075441ab858aaa59f83f157 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:25:15.474578 22454 raft_consensus.cc:504] T 7b19ddec564742ef97c261abb16bb4af P 88ef4913f075441ab858aaa59f83f157 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42ca3747fe9c4b2bbe4bbf4890011742" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 33153 } } peers { permanent_uuid: "88ef4913f075441ab858aaa59f83f157" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 40769 } } peers { permanent_uuid: "404b1ee2892c472d90cd200ff9d136a0" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 42163 } }
I20260928 12:25:15.474888 22454 leader_election.cc:288] T 7b19ddec564742ef97c261abb16bb4af P 88ef4913f075441ab858aaa59f83f157 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42ca3747fe9c4b2bbe4bbf4890011742 (127.2.56.3:33153), 404b1ee2892c472d90cd200ff9d136a0 (127.2.56.2:42163)
I20260928 12:25:15.477473 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:15.475115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:37377 (local address 127.2.56.3:33153)
0928 12:25:15.475162 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:15.475164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:15.475175 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:25:15.475253 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:15.475254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:15.475264 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:15.475333 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:25:15.475484 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:15.475951 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:15.476827 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:15.477162 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:15.477224 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:15.477234 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:15.477373 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 12:25:15.477407 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 12:25:15.477473 22174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:15.475061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.3:33153 (local address 127.2.56.1:37377)
0928 12:25:15.475110 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:15.475122 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:15.475167 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:15.475339 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:15.475341 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:15.475343 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:25:15.475425 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:15.475428 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:15.476019 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:15.476021 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:15.476750 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:15.476752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:15.477226 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:15.477227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:15.477315 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:15.477318 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:25:15.477362 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 12:25:15.477403 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 12:25:15.477591 22447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:15.475266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.56.1:60375 (local address 127.2.56.2:42163)
0928 12:25:15.475321 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:15.475322 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:15.475334 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:25:15.475400 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:15.475417 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:15.475427 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:15.475467 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:25:15.475598 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:15.476044 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:15.476913 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:15.477297 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:15.477356 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:15.477365 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:15.477497 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 12:25:15.477532 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 12:25:15.477595 22457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:15.475151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.2:42163 (local address 127.2.56.1:60375)
0928 12:25:15.475295 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:15.475303 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:15.475342 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:15.475471 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:15.475472 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:15.475473 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:25:15.475538 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:15.475540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:15.476109 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:15.476110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:15.476832 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:15.476833 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:15.477357 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:15.477358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:15.477433 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:15.477436 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:25:15.477479 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 12:25:15.477514 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":104,"threads_started":1}
I20260928 12:25:15.477916 22257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b19ddec564742ef97c261abb16bb4af" candidate_uuid: "88ef4913f075441ab858aaa59f83f157" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "404b1ee2892c472d90cd200ff9d136a0" is_pre_election: true
I20260928 12:25:15.477941 22388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b19ddec564742ef97c261abb16bb4af" candidate_uuid: "88ef4913f075441ab858aaa59f83f157" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42ca3747fe9c4b2bbe4bbf4890011742" is_pre_election: true
I20260928 12:25:15.478036 22257 raft_consensus.cc:2454] T 7b19ddec564742ef97c261abb16bb4af P 404b1ee2892c472d90cd200ff9d136a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88ef4913f075441ab858aaa59f83f157 in term 0.
I20260928 12:25:15.478071 22388 raft_consensus.cc:2454] T 7b19ddec564742ef97c261abb16bb4af P 42ca3747fe9c4b2bbe4bbf4890011742 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88ef4913f075441ab858aaa59f83f157 in term 0.
I20260928 12:25:15.478266 22059 leader_election.cc:302] T 7b19ddec564742ef97c261abb16bb4af P 88ef4913f075441ab858aaa59f83f157 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 404b1ee2892c472d90cd200ff9d136a0, 88ef4913f075441ab858aaa59f83f157; no voters:
I20260928 12:25:15.478395 22454 raft_consensus.cc:2781] T 7b19ddec564742ef97c261abb16bb4af P 88ef4913f075441ab858aaa59f83f157 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:25:15.478449 22454 raft_consensus.cc:482] T 7b19ddec564742ef97c261abb16bb4af P 88ef4913f075441ab858aaa59f83f157 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:25:15.478471 22454 raft_consensus.cc:3037] T 7b19ddec564742ef97c261abb16bb4af P 88ef4913f075441ab858aaa59f83f157 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:15.479125 22454 raft_consensus.cc:504] T 7b19ddec564742ef97c261abb16bb4af P 88ef4913f075441ab858aaa59f83f157 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42ca3747fe9c4b2bbe4bbf4890011742" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 33153 } } peers { permanent_uuid: "88ef4913f075441ab858aaa59f83f157" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 40769 } } peers { permanent_uuid: "404b1ee2892c472d90cd200ff9d136a0" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 42163 } }
I20260928 12:25:15.479238 22454 leader_election.cc:288] T 7b19ddec564742ef97c261abb16bb4af P 88ef4913f075441ab858aaa59f83f157 [CANDIDATE]: Term 1 election: Requested vote from peers 42ca3747fe9c4b2bbe4bbf4890011742 (127.2.56.3:33153), 404b1ee2892c472d90cd200ff9d136a0 (127.2.56.2:42163)
I20260928 12:25:15.479389 22388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b19ddec564742ef97c261abb16bb4af" candidate_uuid: "88ef4913f075441ab858aaa59f83f157" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42ca3747fe9c4b2bbe4bbf4890011742"
I20260928 12:25:15.479422 22257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b19ddec564742ef97c261abb16bb4af" candidate_uuid: "88ef4913f075441ab858aaa59f83f157" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "404b1ee2892c472d90cd200ff9d136a0"
I20260928 12:25:15.479467 22388 raft_consensus.cc:3037] T 7b19ddec564742ef97c261abb16bb4af P 42ca3747fe9c4b2bbe4bbf4890011742 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:15.479478 22257 raft_consensus.cc:3037] T 7b19ddec564742ef97c261abb16bb4af P 404b1ee2892c472d90cd200ff9d136a0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:15.480293 22257 raft_consensus.cc:2454] T 7b19ddec564742ef97c261abb16bb4af P 404b1ee2892c472d90cd200ff9d136a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88ef4913f075441ab858aaa59f83f157 in term 1.
I20260928 12:25:15.480293 22388 raft_consensus.cc:2454] T 7b19ddec564742ef97c261abb16bb4af P 42ca3747fe9c4b2bbe4bbf4890011742 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88ef4913f075441ab858aaa59f83f157 in term 1.
I20260928 12:25:15.480472 22059 leader_election.cc:302] T 7b19ddec564742ef97c261abb16bb4af P 88ef4913f075441ab858aaa59f83f157 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 404b1ee2892c472d90cd200ff9d136a0, 88ef4913f075441ab858aaa59f83f157; no voters:
I20260928 12:25:15.480545 22454 raft_consensus.cc:2781] T 7b19ddec564742ef97c261abb16bb4af P 88ef4913f075441ab858aaa59f83f157 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:25:15.480690 22454 raft_consensus.cc:686] T 7b19ddec564742ef97c261abb16bb4af P 88ef4913f075441ab858aaa59f83f157 [term 1 LEADER]: Becoming Leader. State: Replica: 88ef4913f075441ab858aaa59f83f157, State: Running, Role: LEADER
I20260928 12:25:15.480777 22454 consensus_queue.cc:237] T 7b19ddec564742ef97c261abb16bb4af P 88ef4913f075441ab858aaa59f83f157 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42ca3747fe9c4b2bbe4bbf4890011742" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 33153 } } peers { permanent_uuid: "88ef4913f075441ab858aaa59f83f157" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 40769 } } peers { permanent_uuid: "404b1ee2892c472d90cd200ff9d136a0" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 42163 } }
I20260928 12:25:15.481613 21984 catalog_manager.cc:5495] T 7b19ddec564742ef97c261abb16bb4af P 88ef4913f075441ab858aaa59f83f157 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88ef4913f075441ab858aaa59f83f157 (127.2.56.1). New cstate: current_term: 1 leader_uuid: "88ef4913f075441ab858aaa59f83f157" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42ca3747fe9c4b2bbe4bbf4890011742" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 33153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88ef4913f075441ab858aaa59f83f157" member_type: VOTER last_known_addr { host: "127.2.56.1" port: 40769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "404b1ee2892c472d90cd200ff9d136a0" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 42163 } health_report { overall_health: UNKNOWN } } }
I20260928 12:25:15.496013 22444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:15.493961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33400 (local address 127.2.56.1:40769)
0928 12:25:15.494011 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:15.494013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:15.494021 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:25:15.494070 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:15.494071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:15.494087 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:15.494140 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:25:15.494237 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:15.494690 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:15.495378 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:15.495574 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:15.495632 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:15.495737 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:25:15.495827 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:25:15.495885 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:15.495897 (+ 12us) server_negotiation.cc:299] Negotiation successful
0928 12:25:15.495912 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 12:25:15.497685 22257 raft_consensus.cc:1261] T 7b19ddec564742ef97c261abb16bb4af P 404b1ee2892c472d90cd200ff9d136a0 [term 1 FOLLOWER]: Refusing update from remote peer 88ef4913f075441ab858aaa59f83f157: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:25:15.497700 22388 raft_consensus.cc:1261] T 7b19ddec564742ef97c261abb16bb4af P 42ca3747fe9c4b2bbe4bbf4890011742 [term 1 FOLLOWER]: Refusing update from remote peer 88ef4913f075441ab858aaa59f83f157: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:25:15.498008 22458 consensus_queue.cc:1035] T 7b19ddec564742ef97c261abb16bb4af P 88ef4913f075441ab858aaa59f83f157 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42ca3747fe9c4b2bbe4bbf4890011742" member_type: VOTER last_known_addr { host: "127.2.56.3" port: 33153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:25:15.498134 22454 consensus_queue.cc:1035] T 7b19ddec564742ef97c261abb16bb4af P 88ef4913f075441ab858aaa59f83f157 [LEADER]: Connected to new peer: Peer: permanent_uuid: "404b1ee2892c472d90cd200ff9d136a0" member_type: VOTER last_known_addr { host: "127.2.56.2" port: 42163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:25:15.500162 22461 mvcc.cc:205] Tried to move back new op lower bound from 7334290700275912704 to 7334290700209889280. Current Snapshot: MvccSnapshot[applied={T|T < 7334290700275912704}]
I20260928 12:25:15.500335 22463 mvcc.cc:205] Tried to move back new op lower bound from 7334290700275912704 to 7334290700209889280. Current Snapshot: MvccSnapshot[applied={T|T < 7334290700275912704}]
I20260928 12:25:15.500825 22462 mvcc.cc:205] Tried to move back new op lower bound from 7334290700275912704 to 7334290700209889280. Current Snapshot: MvccSnapshot[applied={T|T < 7334290700275912704}]
I20260928 12:25:15.502410 21984 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:25:15.502491 21984 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:25:15.503100 21984 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 15858f7da4af4604a8b69562bd9109cb: Sending DeleteTablet for 3 replicas of tablet 7b19ddec564742ef97c261abb16bb4af
I20260928 12:25:15.503435 22237 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b19ddec564742ef97c261abb16bb4af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:25:15 UTC) from {username='slave'} at 127.0.0.1:52322
I20260928 12:25:15.503422 22368 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b19ddec564742ef97c261abb16bb4af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:25:15 UTC) from {username='slave'} at 127.0.0.1:55450
I20260928 12:25:15.503618 22106 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b19ddec564742ef97c261abb16bb4af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:25:15 UTC) from {username='slave'} at 127.0.0.1:33390
I20260928 12:25:15.503714 22471 tablet_replica.cc:330] T 7b19ddec564742ef97c261abb16bb4af P 404b1ee2892c472d90cd200ff9d136a0: stopping tablet replica
I20260928 12:25:15.503803 22471 raft_consensus.cc:2229] T 7b19ddec564742ef97c261abb16bb4af P 404b1ee2892c472d90cd200ff9d136a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:25:15.503895 22471 raft_consensus.cc:2258] T 7b19ddec564742ef97c261abb16bb4af P 404b1ee2892c472d90cd200ff9d136a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:25:15.504106 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 22044
I20260928 12:25:15.504292 22471 ts_tablet_manager.cc:1905] T 7b19ddec564742ef97c261abb16bb4af P 404b1ee2892c472d90cd200ff9d136a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:25:15.505980 22471 ts_tablet_manager.cc:1918] T 7b19ddec564742ef97c261abb16bb4af P 404b1ee2892c472d90cd200ff9d136a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:25:15.506065 22471 log.cc:1196] T 7b19ddec564742ef97c261abb16bb4af P 404b1ee2892c472d90cd200ff9d136a0: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-1/wal/wals/7b19ddec564742ef97c261abb16bb4af
I20260928 12:25:15.506291 22471 ts_tablet_manager.cc:1939] T 7b19ddec564742ef97c261abb16bb4af P 404b1ee2892c472d90cd200ff9d136a0: Deleting consensus metadata
I20260928 12:25:15.506639 21968 catalog_manager.cc:4841] TS 404b1ee2892c472d90cd200ff9d136a0 (127.2.56.2:42163): tablet 7b19ddec564742ef97c261abb16bb4af (table test-table [id=aae07935fe9441a9b9f2bfce8095c149]) successfully deleted
W20260928 12:25:15.510027 21970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.56.1:40769 (error 108)
I20260928 12:25:15.510073 22472 tablet_replica.cc:330] T 7b19ddec564742ef97c261abb16bb4af P 42ca3747fe9c4b2bbe4bbf4890011742: stopping tablet replica
I20260928 12:25:15.510154 22472 raft_consensus.cc:2229] T 7b19ddec564742ef97c261abb16bb4af P 42ca3747fe9c4b2bbe4bbf4890011742 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:25:15.510210 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 22175
I20260928 12:25:15.510251 22472 raft_consensus.cc:2258] T 7b19ddec564742ef97c261abb16bb4af P 42ca3747fe9c4b2bbe4bbf4890011742 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:25:15.511615 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:15.511488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.1:40769 (local address 127.0.0.1:33408)
0928 12:25:15.511540 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:15.511590 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.1:40769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 12:25:15.511721 21970 catalog_manager.cc:4584] TS 88ef4913f075441ab858aaa59f83f157 (127.2.56.1:40769): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7b19ddec564742ef97c261abb16bb4af: Network error: Client connection negotiation failed: client connection to 127.2.56.1:40769: connect: Connection refused (error 111)
I20260928 12:25:15.512146 22472 ts_tablet_manager.cc:1905] T 7b19ddec564742ef97c261abb16bb4af P 42ca3747fe9c4b2bbe4bbf4890011742: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:25:15.513610 22472 ts_tablet_manager.cc:1918] T 7b19ddec564742ef97c261abb16bb4af P 42ca3747fe9c4b2bbe4bbf4890011742: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:25:15.513681 22472 log.cc:1196] T 7b19ddec564742ef97c261abb16bb4af P 42ca3747fe9c4b2bbe4bbf4890011742: Deleting WAL directory at /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598072638673-2272-0/minicluster-data/ts-2/wal/wals/7b19ddec564742ef97c261abb16bb4af
I20260928 12:25:15.513895 22472 ts_tablet_manager.cc:1939] T 7b19ddec564742ef97c261abb16bb4af P 42ca3747fe9c4b2bbe4bbf4890011742: Deleting consensus metadata
I20260928 12:25:15.514241 21968 catalog_manager.cc:4841] TS 42ca3747fe9c4b2bbe4bbf4890011742 (127.2.56.3:33153): tablet 7b19ddec564742ef97c261abb16bb4af (table test-table [id=aae07935fe9441a9b9f2bfce8095c149]) successfully deleted
I20260928 12:25:15.515043 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 22306
I20260928 12:25:15.518868 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 21953
2026-09-28T12:25:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (763 ms)
[----------] 6 tests from EncryptionPolicyTest (7670 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:25:15 dist-test-slave-920p krb5kdc[22477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:25:15 dist-test-slave-920p krb5kdc[22477](info): set up 2 sockets
Sep 28 12:25:15 dist-test-slave-920p krb5kdc[22477](info): commencing operation
krb5kdc: starting...
W20260928 12:25:16.468995 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:25:16 dist-test-slave-920p krb5kdc[22477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:25:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:25:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 12:25:16.576953 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39953
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:25:16.655452 22490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:16.655655 22490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:16.655678 22490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:16.656991 22490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:25:16.657043 22490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:16.657058 22490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:16.657069 22490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:25:16.657080 22490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:25:16.658595 22490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39953
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:16.658792 22490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:16.658964 22490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:16.663486 22496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:16.663470 22495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:16.663641 22490 server_base.cc:1034] running on GCE node
W20260928 12:25:16.663506 22498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:16.663932 22490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:16.664189 22490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:16.665314 22490 hybrid_clock.cc:648] HybridClock initialized: now 1790598316665301 us; error 27 us; skew 500 ppm
Sep 28 12:25:16 dist-test-slave-920p krb5kdc[22477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:16.667641 22490 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 12:25:16.668038 22490 webserver.cc:460] Webserver started at http://127.0.0.1:35365/ using document root <none> and password file <none>
I20260928 12:25:16.668226 22490 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:16.668272 22490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:16.668387 22490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:16.669410 22490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "eaa1dd39658441218ea3e96bfdd32036"
format_stamp: "Formatted at 2026-09-28 12:25:16 on dist-test-slave-920p"
server_key: "1a76a6909d2cfcaf79ac71ea628aae57"
server_key_iv: "41c0de4a5111508752f9a2b87773eb4a"
server_key_version: "encryptionkey@0"
I20260928 12:25:16.669727 22490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "eaa1dd39658441218ea3e96bfdd32036"
format_stamp: "Formatted at 2026-09-28 12:25:16 on dist-test-slave-920p"
server_key: "1a76a6909d2cfcaf79ac71ea628aae57"
server_key_iv: "41c0de4a5111508752f9a2b87773eb4a"
server_key_version: "encryptionkey@0"
I20260928 12:25:16.670881 22490 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:25:16.671526 22505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:16.671684 22490 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:25:16.671775 22490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "eaa1dd39658441218ea3e96bfdd32036"
format_stamp: "Formatted at 2026-09-28 12:25:16 on dist-test-slave-920p"
server_key: "1a76a6909d2cfcaf79ac71ea628aae57"
server_key_iv: "41c0de4a5111508752f9a2b87773eb4a"
server_key_version: "encryptionkey@0"
I20260928 12:25:16.671832 22490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:16.682090 22490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:16.683945 22490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:16.684063 22490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:16.687613 22490 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39953
I20260928 12:25:16.687644 22557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39953 every 8 connection(s)
I20260928 12:25:16.687965 22490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:25:16.688834 22558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:16.690980 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 22490
I20260928 12:25:16.691032 22558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaa1dd39658441218ea3e96bfdd32036: Bootstrap starting.
I20260928 12:25:16.691084 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:25:16.691238 2272 external_mini_cluster.cc:1428] Setting key 305c8cbab706d68553865bc048a0847d
I20260928 12:25:16.691640 22558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eaa1dd39658441218ea3e96bfdd32036: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:16.691969 22558 log.cc:824] T 00000000000000000000000000000000 P eaa1dd39658441218ea3e96bfdd32036: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:16.692615 22558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaa1dd39658441218ea3e96bfdd32036: No bootstrap required, opened a new log
I20260928 12:25:16.694669 22558 raft_consensus.cc:348] T 00000000000000000000000000000000 P eaa1dd39658441218ea3e96bfdd32036 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa1dd39658441218ea3e96bfdd32036" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39953 } }
I20260928 12:25:16.694828 22558 raft_consensus.cc:374] T 00000000000000000000000000000000 P eaa1dd39658441218ea3e96bfdd32036 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:16.694856 22558 raft_consensus.cc:729] T 00000000000000000000000000000000 P eaa1dd39658441218ea3e96bfdd32036 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaa1dd39658441218ea3e96bfdd32036, State: Initialized, Role: FOLLOWER
I20260928 12:25:16.694967 22558 consensus_queue.cc:260] T 00000000000000000000000000000000 P eaa1dd39658441218ea3e96bfdd32036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa1dd39658441218ea3e96bfdd32036" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39953 } }
I20260928 12:25:16.695027 22558 raft_consensus.cc:388] T 00000000000000000000000000000000 P eaa1dd39658441218ea3e96bfdd32036 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:25:16.695060 22558 raft_consensus.cc:482] T 00000000000000000000000000000000 P eaa1dd39658441218ea3e96bfdd32036 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:25:16.695102 22558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eaa1dd39658441218ea3e96bfdd32036 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:16.695574 22558 raft_consensus.cc:504] T 00000000000000000000000000000000 P eaa1dd39658441218ea3e96bfdd32036 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa1dd39658441218ea3e96bfdd32036" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39953 } }
I20260928 12:25:16.695677 22558 leader_election.cc:302] T 00000000000000000000000000000000 P eaa1dd39658441218ea3e96bfdd32036 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eaa1dd39658441218ea3e96bfdd32036; no voters:
I20260928 12:25:16.695859 22558 leader_election.cc:288] T 00000000000000000000000000000000 P eaa1dd39658441218ea3e96bfdd32036 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:25:16.695891 22563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eaa1dd39658441218ea3e96bfdd32036 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:25:16.696029 22563 raft_consensus.cc:686] T 00000000000000000000000000000000 P eaa1dd39658441218ea3e96bfdd32036 [term 1 LEADER]: Becoming Leader. State: Replica: eaa1dd39658441218ea3e96bfdd32036, State: Running, Role: LEADER
I20260928 12:25:16.696105 22563 consensus_queue.cc:237] T 00000000000000000000000000000000 P eaa1dd39658441218ea3e96bfdd32036 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa1dd39658441218ea3e96bfdd32036" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39953 } }
I20260928 12:25:16.696107 22558 sys_catalog.cc:564] T 00000000000000000000000000000000 P eaa1dd39658441218ea3e96bfdd32036 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:25:16.696410 22563 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaa1dd39658441218ea3e96bfdd32036 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eaa1dd39658441218ea3e96bfdd32036. Latest consensus state: current_term: 1 leader_uuid: "eaa1dd39658441218ea3e96bfdd32036" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa1dd39658441218ea3e96bfdd32036" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39953 } } }
I20260928 12:25:16.696549 22563 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaa1dd39658441218ea3e96bfdd32036 [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:16.697397 22569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:25:16.697470 22564 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaa1dd39658441218ea3e96bfdd32036 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eaa1dd39658441218ea3e96bfdd32036" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa1dd39658441218ea3e96bfdd32036" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39953 } } }
I20260928 12:25:16.697571 22564 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaa1dd39658441218ea3e96bfdd32036 [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:16.697861 22569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 12:25:16 dist-test-slave-920p krb5kdc[22477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 12:25:16.699484 22569 catalog_manager.cc:1347] Generated new cluster ID: 89995f538033431a9eb560f636162d59
I20260928 12:25:16.699534 22569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:25:16.702000 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:16.691729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49807 (local address 127.0.0.1:39953)
0928 12:25:16.691889 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:16.691896 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:16.691920 (+ 24us) server_negotiation.cc:407] Connection header received
0928 12:25:16.692433 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:16.692439 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:16.692485 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:16.692593 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:16.693349 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:16.694181 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:16.695206 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:16.695591 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:16.695668 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:16.698409 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:16.698412 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:16.698413 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:16.698422 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:16.700240 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:16.700641 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:16.700644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:16.700644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:16.700753 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:16.700965 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:16.700966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:16.700966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:16.701286 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:16.701437 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:16.701711 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 12:25:16.701805 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":64,"threads_started":1}
I20260928 12:25:16.702894 2272 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:25:16.708562 22569 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:25:16.709182 22569 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:25:16.710938 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:16.704454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42323 (local address 127.0.0.1:39953)
0928 12:25:16.704481 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:16.704483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:16.704490 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:25:16.704527 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:16.704527 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:16.704535 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:16.704576 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:16.705165 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:16.705881 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:16.706834 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:16.707154 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:16.707207 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:16.708486 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:16.708487 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:16.708487 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:16.708493 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:16.709896 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:16.710274 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:16.710275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:16.710275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:16.710321 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:16.710503 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:16.710504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:16.710504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:16.710601 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:16.710677 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:16.710804 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 12:25:16.710858 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260928 12:25:16.714959 22569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eaa1dd39658441218ea3e96bfdd32036: Generated new TSK 0
I20260928 12:25:16.715111 22569 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:25:16.717061 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 22490
W20260928 12:25:16.720880 22475 connection.cc:535] client connection to 127.0.0.1:39953 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39953): Connection reset by peer (error 104)
2026-09-28T12:25:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1197 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:25:16 dist-test-slave-920p krb5kdc[22589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:25:16 dist-test-slave-920p krb5kdc[22589](info): set up 2 sockets
Sep 28 12:25:16 dist-test-slave-920p krb5kdc[22589](info): commencing operation
krb5kdc: starting...
W20260928 12:25:17.669166 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.003s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:25:17 dist-test-slave-920p krb5kdc[22589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:25:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:25:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 12:25:17.792544 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42797
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:25:17.891950 22602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:17.892196 22602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:17.892234 22602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:17.893537 22602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:25:17.893589 22602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:17.893605 22602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:17.893623 22602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:25:17.893638 22602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:25:17.895219 22602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42797
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:17.895483 22602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:17.895699 22602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:17.900580 22608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:17.900588 22607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:17.900695 22602 server_base.cc:1034] running on GCE node
W20260928 12:25:17.900594 22610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:17.901026 22602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:17.901320 22602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:17.902458 22602 hybrid_clock.cc:648] HybridClock initialized: now 1790598317902441 us; error 35 us; skew 500 ppm
Sep 28 12:25:17 dist-test-slave-920p krb5kdc[22589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:17.904836 22602 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 12:25:17.905318 22602 webserver.cc:460] Webserver started at http://127.0.0.1:43097/ using document root <none> and password file <none>
I20260928 12:25:17.905530 22602 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:17.905575 22602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:17.905684 22602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:17.906764 22602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "6aaf011e86c7444f91151884816b1014"
format_stamp: "Formatted at 2026-09-28 12:25:17 on dist-test-slave-920p"
server_key: "710711cd1b8eb4bc80fde98e066d58bf"
server_key_iv: "f5c695f7fb8f7fb51bdbf6cb2023d07e"
server_key_version: "encryptionkey@0"
I20260928 12:25:17.907092 22602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "6aaf011e86c7444f91151884816b1014"
format_stamp: "Formatted at 2026-09-28 12:25:17 on dist-test-slave-920p"
server_key: "710711cd1b8eb4bc80fde98e066d58bf"
server_key_iv: "f5c695f7fb8f7fb51bdbf6cb2023d07e"
server_key_version: "encryptionkey@0"
I20260928 12:25:17.908335 22602 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 12:25:17.909054 22617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:17.909219 22602 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:25:17.909327 22602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "6aaf011e86c7444f91151884816b1014"
format_stamp: "Formatted at 2026-09-28 12:25:17 on dist-test-slave-920p"
server_key: "710711cd1b8eb4bc80fde98e066d58bf"
server_key_iv: "f5c695f7fb8f7fb51bdbf6cb2023d07e"
server_key_version: "encryptionkey@0"
I20260928 12:25:17.909395 22602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:17.935075 22602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:17.937003 22602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:17.937129 22602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:17.940946 22602 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42797
I20260928 12:25:17.940989 22669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42797 every 8 connection(s)
I20260928 12:25:17.941324 22602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:25:17.942253 22670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:17.944475 22670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aaf011e86c7444f91151884816b1014: Bootstrap starting.
I20260928 12:25:17.945027 22670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6aaf011e86c7444f91151884816b1014: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:17.945309 22670 log.cc:824] T 00000000000000000000000000000000 P 6aaf011e86c7444f91151884816b1014: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:17.945941 22670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aaf011e86c7444f91151884816b1014: No bootstrap required, opened a new log
I20260928 12:25:17.947525 22670 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6aaf011e86c7444f91151884816b1014 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aaf011e86c7444f91151884816b1014" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42797 } }
I20260928 12:25:17.947685 22670 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6aaf011e86c7444f91151884816b1014 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:17.947718 22670 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6aaf011e86c7444f91151884816b1014 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aaf011e86c7444f91151884816b1014, State: Initialized, Role: FOLLOWER
I20260928 12:25:17.947840 22670 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6aaf011e86c7444f91151884816b1014 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aaf011e86c7444f91151884816b1014" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42797 } }
I20260928 12:25:17.947901 22670 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6aaf011e86c7444f91151884816b1014 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:25:17.947929 22670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6aaf011e86c7444f91151884816b1014 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:25:17.947974 22670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6aaf011e86c7444f91151884816b1014 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:17.948318 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 22602
I20260928 12:25:17.948402 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:25:17.948530 22670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6aaf011e86c7444f91151884816b1014 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aaf011e86c7444f91151884816b1014" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42797 } }
I20260928 12:25:17.948557 2272 external_mini_cluster.cc:1428] Setting key 5b2d3be731a49e96aad7c3a42c477295
I20260928 12:25:17.948633 22670 leader_election.cc:302] T 00000000000000000000000000000000 P 6aaf011e86c7444f91151884816b1014 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6aaf011e86c7444f91151884816b1014; no voters:
I20260928 12:25:17.948781 22670 leader_election.cc:288] T 00000000000000000000000000000000 P 6aaf011e86c7444f91151884816b1014 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:25:17.948820 22673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6aaf011e86c7444f91151884816b1014 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:25:17.948931 22673 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6aaf011e86c7444f91151884816b1014 [term 1 LEADER]: Becoming Leader. State: Replica: 6aaf011e86c7444f91151884816b1014, State: Running, Role: LEADER
I20260928 12:25:17.949036 22670 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6aaf011e86c7444f91151884816b1014 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:25:17.949069 22673 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6aaf011e86c7444f91151884816b1014 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aaf011e86c7444f91151884816b1014" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42797 } }
I20260928 12:25:17.949553 22675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aaf011e86c7444f91151884816b1014 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6aaf011e86c7444f91151884816b1014" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aaf011e86c7444f91151884816b1014" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42797 } } }
I20260928 12:25:17.949694 22675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aaf011e86c7444f91151884816b1014 [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:17.949685 22674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aaf011e86c7444f91151884816b1014 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6aaf011e86c7444f91151884816b1014. Latest consensus state: current_term: 1 leader_uuid: "6aaf011e86c7444f91151884816b1014" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aaf011e86c7444f91151884816b1014" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42797 } } }
I20260928 12:25:17.949962 22674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aaf011e86c7444f91151884816b1014 [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:17.950318 22684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:25:17.950841 22684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:25:17.952826 22684 catalog_manager.cc:1347] Generated new cluster ID: 430b2000a3684db18a3a458f7243ee33
I20260928 12:25:17.952876 22684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:25:17.960760 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:17.949142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57275 (local address 127.0.0.1:42797)
0928 12:25:17.949542 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:17.949552 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:17.949588 (+ 36us) server_negotiation.cc:407] Connection header received
0928 12:25:17.949721 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:17.949729 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:17.949790 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:17.950205 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:17.950327 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:17.951415 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:17.952486 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:17.952986 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:17.953078 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:17.960073 (+ 6995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:17.960078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:17.960078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:17.960287 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:17.960322 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:17.960335 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:17.960413 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:17.960528 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 12:25:17.960561 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":89,"threads_started":1}
I20260928 12:25:17.961721 2272 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:25:17.968744 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:17.965536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38659 (local address 127.0.0.1:42797)
0928 12:25:17.965582 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:17.965584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:17.965760 (+ 176us) server_negotiation.cc:407] Connection header received
0928 12:25:17.965817 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:17.965818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:17.965828 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:17.965869 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:17.965989 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:17.966633 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:17.967454 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:17.967666 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:17.967723 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:17.968443 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:17.968445 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:17.968445 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:17.968450 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:17.968464 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:17.968468 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:17.968514 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:17.968646 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 12:25:17.968660 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 12:25:17.974614 22684 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:25:17.975194 22684 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:25:17.983260 22684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6aaf011e86c7444f91151884816b1014: Generated new TSK 0
I20260928 12:25:17.983464 22684 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:25:17.994612 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 22602
W20260928 12:25:17.998335 22587 connection.cc:535] client connection to 127.0.0.1:42797 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42797): Connection reset by peer (error 104)
2026-09-28T12:25:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1277 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:25:18 dist-test-slave-920p krb5kdc[22701](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:25:18 dist-test-slave-920p krb5kdc[22701](info): set up 2 sockets
Sep 28 12:25:18 dist-test-slave-920p krb5kdc[22701](info): commencing operation
krb5kdc: starting...
W20260928 12:25:18.957176 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:25:19 dist-test-slave-920p krb5kdc[22701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:25:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:25:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 12:25:19.068228 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44823
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:25:19.146991 22714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:19.147193 22714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:19.147223 22714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:19.148351 22714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:25:19.148389 22714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:19.148408 22714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:19.148424 22714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:25:19.148439 22714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:25:19.149874 22714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44823
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:19.150107 22714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:19.150301 22714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:19.154740 22722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:19.154769 22720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:19.154837 22714 server_base.cc:1034] running on GCE node
W20260928 12:25:19.154783 22719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:19.155143 22714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:19.155404 22714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:19.156533 22714 hybrid_clock.cc:648] HybridClock initialized: now 1790598319156521 us; error 27 us; skew 500 ppm
Sep 28 12:25:19 dist-test-slave-920p krb5kdc[22701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:19.158797 22714 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 12:25:19.159175 22714 webserver.cc:460] Webserver started at http://127.0.0.1:33393/ using document root <none> and password file <none>
I20260928 12:25:19.159370 22714 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:19.159411 22714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:19.159518 22714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:19.160465 22714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "ecc3ca7e52ef44bd8875548c34cac11a"
format_stamp: "Formatted at 2026-09-28 12:25:19 on dist-test-slave-920p"
server_key: "8b91f2cf48fa1cfcbd22bffa7d21cc2a"
server_key_iv: "a705af5e93360966bc5c6966bebb2996"
server_key_version: "encryptionkey@0"
I20260928 12:25:19.160772 22714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "ecc3ca7e52ef44bd8875548c34cac11a"
format_stamp: "Formatted at 2026-09-28 12:25:19 on dist-test-slave-920p"
server_key: "8b91f2cf48fa1cfcbd22bffa7d21cc2a"
server_key_iv: "a705af5e93360966bc5c6966bebb2996"
server_key_version: "encryptionkey@0"
I20260928 12:25:19.161973 22714 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 12:25:19.162609 22729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:19.162761 22714 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:25:19.162838 22714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "ecc3ca7e52ef44bd8875548c34cac11a"
format_stamp: "Formatted at 2026-09-28 12:25:19 on dist-test-slave-920p"
server_key: "8b91f2cf48fa1cfcbd22bffa7d21cc2a"
server_key_iv: "a705af5e93360966bc5c6966bebb2996"
server_key_version: "encryptionkey@0"
I20260928 12:25:19.162894 22714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:19.165936 22714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:19.167814 22714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:19.167932 22714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:19.171654 22714 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44823
I20260928 12:25:19.171689 22781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44823 every 8 connection(s)
I20260928 12:25:19.171993 22714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:25:19.172889 22782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:19.174947 22782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecc3ca7e52ef44bd8875548c34cac11a: Bootstrap starting.
I20260928 12:25:19.175429 22782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecc3ca7e52ef44bd8875548c34cac11a: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:19.175647 22782 log.cc:824] T 00000000000000000000000000000000 P ecc3ca7e52ef44bd8875548c34cac11a: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:19.176187 22782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecc3ca7e52ef44bd8875548c34cac11a: No bootstrap required, opened a new log
I20260928 12:25:19.177702 22782 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecc3ca7e52ef44bd8875548c34cac11a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecc3ca7e52ef44bd8875548c34cac11a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44823 } }
I20260928 12:25:19.177829 22782 raft_consensus.cc:374] T 00000000000000000000000000000000 P ecc3ca7e52ef44bd8875548c34cac11a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:19.177850 22782 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecc3ca7e52ef44bd8875548c34cac11a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc3ca7e52ef44bd8875548c34cac11a, State: Initialized, Role: FOLLOWER
I20260928 12:25:19.177947 22782 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecc3ca7e52ef44bd8875548c34cac11a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecc3ca7e52ef44bd8875548c34cac11a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44823 } }
I20260928 12:25:19.178015 22782 raft_consensus.cc:388] T 00000000000000000000000000000000 P ecc3ca7e52ef44bd8875548c34cac11a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:25:19.178045 22782 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecc3ca7e52ef44bd8875548c34cac11a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:25:19.178076 22782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecc3ca7e52ef44bd8875548c34cac11a [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:19.178524 22782 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecc3ca7e52ef44bd8875548c34cac11a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecc3ca7e52ef44bd8875548c34cac11a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44823 } }
I20260928 12:25:19.178603 22782 leader_election.cc:302] T 00000000000000000000000000000000 P ecc3ca7e52ef44bd8875548c34cac11a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecc3ca7e52ef44bd8875548c34cac11a; no voters:
I20260928 12:25:19.178715 22782 leader_election.cc:288] T 00000000000000000000000000000000 P ecc3ca7e52ef44bd8875548c34cac11a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:25:19.178761 22785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecc3ca7e52ef44bd8875548c34cac11a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:25:19.178912 22782 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecc3ca7e52ef44bd8875548c34cac11a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:25:19.179116 22785 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecc3ca7e52ef44bd8875548c34cac11a [term 1 LEADER]: Becoming Leader. State: Replica: ecc3ca7e52ef44bd8875548c34cac11a, State: Running, Role: LEADER
I20260928 12:25:19.179199 22785 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecc3ca7e52ef44bd8875548c34cac11a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecc3ca7e52ef44bd8875548c34cac11a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44823 } }
I20260928 12:25:19.179534 22787 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecc3ca7e52ef44bd8875548c34cac11a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecc3ca7e52ef44bd8875548c34cac11a. Latest consensus state: current_term: 1 leader_uuid: "ecc3ca7e52ef44bd8875548c34cac11a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecc3ca7e52ef44bd8875548c34cac11a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44823 } } }
I20260928 12:25:19.179533 22786 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecc3ca7e52ef44bd8875548c34cac11a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ecc3ca7e52ef44bd8875548c34cac11a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecc3ca7e52ef44bd8875548c34cac11a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44823 } } }
I20260928 12:25:19.179639 22787 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecc3ca7e52ef44bd8875548c34cac11a [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:19.179643 22786 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecc3ca7e52ef44bd8875548c34cac11a [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:19.179834 22794 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:25:19.180298 22794 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:25:19.181669 22794 catalog_manager.cc:1347] Generated new cluster ID: 6decbb2a4c3b4f29b7ce5422e482d3b5
I20260928 12:25:19.181720 22794 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:25:19.182199 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 22714
I20260928 12:25:19.182292 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:25:19.182458 2272 external_mini_cluster.cc:1428] Setting key a1bbd8e562d036d6970895d0570be600
I20260928 12:25:19.187184 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:19.182869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49579 (local address 127.0.0.1:44823)
0928 12:25:19.183040 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:19.183049 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:19.185476 (+ 2427us) server_negotiation.cc:407] Connection header received
0928 12:25:19.185602 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:19.185609 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:19.185638 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:19.185771 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:19.186689 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:19.186693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:19.186693 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:19.186696 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:19.186723 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:19.186735 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:19.186791 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:19.186864 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 12:25:19.186901 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20260928 12:25:19.188019 2272 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:25:19.190337 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:19.189389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49781 (local address 127.0.0.1:44823)
0928 12:25:19.189418 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:19.189420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:19.189544 (+ 124us) server_negotiation.cc:407] Connection header received
0928 12:25:19.189602 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:19.189603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:19.189608 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:19.189659 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:19.190145 (+ 486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:19.190145 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:19.190146 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:19.190149 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:19.190162 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:19.190166 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:19.190218 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:19.190271 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 12:25:19.190283 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260928 12:25:19.191126 22794 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:25:19.191236 22794 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:25:19.197394 22794 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ecc3ca7e52ef44bd8875548c34cac11a: Generated new TSK 0
I20260928 12:25:19.197547 22794 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:25:19.203085 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 22714
2026-09-28T12:25:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1208 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:25:19 dist-test-slave-920p krb5kdc[22813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:25:19 dist-test-slave-920p krb5kdc[22813](info): set up 2 sockets
Sep 28 12:25:19 dist-test-slave-920p krb5kdc[22813](info): commencing operation
krb5kdc: starting...
W20260928 12:25:20.149135 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:25:20 dist-test-slave-920p krb5kdc[22813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:25:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:25:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 12:25:20.255999 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36683
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:25:20.331993 22826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:20.332178 22826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:20.332208 22826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:20.333319 22826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:25:20.333355 22826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:20.333374 22826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:20.333389 22826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:25:20.333405 22826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:25:20.334800 22826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:20.335019 22826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:20.335206 22826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:20.339562 22832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:20.339564 22834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:20.339663 22826 server_base.cc:1034] running on GCE node
W20260928 12:25:20.339598 22831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:20.340029 22826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:20.340293 22826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:20.341423 22826 hybrid_clock.cc:648] HybridClock initialized: now 1790598320341412 us; error 25 us; skew 500 ppm
Sep 28 12:25:20 dist-test-slave-920p krb5kdc[22813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:20.343606 22826 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 12:25:20.344004 22826 webserver.cc:460] Webserver started at http://127.0.0.1:43837/ using document root <none> and password file <none>
I20260928 12:25:20.344202 22826 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:20.344249 22826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:20.344365 22826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:20.345369 22826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "26a23a4466f14f999d095bf4a41ea7cd"
format_stamp: "Formatted at 2026-09-28 12:25:20 on dist-test-slave-920p"
server_key: "1d25c3a2cbc7dace7f1fd74320e4520e"
server_key_iv: "deb6e997f1c849f56218f17062c36d25"
server_key_version: "encryptionkey@0"
I20260928 12:25:20.345702 22826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "26a23a4466f14f999d095bf4a41ea7cd"
format_stamp: "Formatted at 2026-09-28 12:25:20 on dist-test-slave-920p"
server_key: "1d25c3a2cbc7dace7f1fd74320e4520e"
server_key_iv: "deb6e997f1c849f56218f17062c36d25"
server_key_version: "encryptionkey@0"
I20260928 12:25:20.346868 22826 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:25:20.347446 22841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:20.347599 22826 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:25:20.347673 22826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "26a23a4466f14f999d095bf4a41ea7cd"
format_stamp: "Formatted at 2026-09-28 12:25:20 on dist-test-slave-920p"
server_key: "1d25c3a2cbc7dace7f1fd74320e4520e"
server_key_iv: "deb6e997f1c849f56218f17062c36d25"
server_key_version: "encryptionkey@0"
I20260928 12:25:20.347735 22826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:20.374841 22826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:20.376560 22826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:20.376700 22826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:20.380333 22826 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36683
I20260928 12:25:20.380379 22893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36683 every 8 connection(s)
I20260928 12:25:20.380688 22826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:25:20.381633 22894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:20.384191 22894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26a23a4466f14f999d095bf4a41ea7cd: Bootstrap starting.
I20260928 12:25:20.384773 22894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26a23a4466f14f999d095bf4a41ea7cd: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:20.385026 22894 log.cc:824] T 00000000000000000000000000000000 P 26a23a4466f14f999d095bf4a41ea7cd: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:20.385800 22894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26a23a4466f14f999d095bf4a41ea7cd: No bootstrap required, opened a new log
I20260928 12:25:20.387413 22894 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26a23a4466f14f999d095bf4a41ea7cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26a23a4466f14f999d095bf4a41ea7cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36683 } }
I20260928 12:25:20.387549 22894 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26a23a4466f14f999d095bf4a41ea7cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:20.387570 22894 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26a23a4466f14f999d095bf4a41ea7cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26a23a4466f14f999d095bf4a41ea7cd, State: Initialized, Role: FOLLOWER
I20260928 12:25:20.387696 22894 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26a23a4466f14f999d095bf4a41ea7cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26a23a4466f14f999d095bf4a41ea7cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36683 } }
I20260928 12:25:20.387755 22894 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26a23a4466f14f999d095bf4a41ea7cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:25:20.387776 22894 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26a23a4466f14f999d095bf4a41ea7cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:25:20.387820 22894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26a23a4466f14f999d095bf4a41ea7cd [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:20.388342 22894 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26a23a4466f14f999d095bf4a41ea7cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26a23a4466f14f999d095bf4a41ea7cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36683 } }
I20260928 12:25:20.388432 22894 leader_election.cc:302] T 00000000000000000000000000000000 P 26a23a4466f14f999d095bf4a41ea7cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26a23a4466f14f999d095bf4a41ea7cd; no voters:
I20260928 12:25:20.388595 22894 leader_election.cc:288] T 00000000000000000000000000000000 P 26a23a4466f14f999d095bf4a41ea7cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:25:20.388629 22897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26a23a4466f14f999d095bf4a41ea7cd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:25:20.388783 22897 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26a23a4466f14f999d095bf4a41ea7cd [term 1 LEADER]: Becoming Leader. State: Replica: 26a23a4466f14f999d095bf4a41ea7cd, State: Running, Role: LEADER
I20260928 12:25:20.388839 22894 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26a23a4466f14f999d095bf4a41ea7cd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:25:20.388877 22897 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26a23a4466f14f999d095bf4a41ea7cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26a23a4466f14f999d095bf4a41ea7cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36683 } }
I20260928 12:25:20.389240 22898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26a23a4466f14f999d095bf4a41ea7cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26a23a4466f14f999d095bf4a41ea7cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26a23a4466f14f999d095bf4a41ea7cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36683 } } }
I20260928 12:25:20.389392 22898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26a23a4466f14f999d095bf4a41ea7cd [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:20.389256 22899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26a23a4466f14f999d095bf4a41ea7cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26a23a4466f14f999d095bf4a41ea7cd. Latest consensus state: current_term: 1 leader_uuid: "26a23a4466f14f999d095bf4a41ea7cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26a23a4466f14f999d095bf4a41ea7cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36683 } } }
I20260928 12:25:20.389592 22901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:25:20.389602 22899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26a23a4466f14f999d095bf4a41ea7cd [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:20.389883 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 22826
I20260928 12:25:20.389964 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:25:20.390118 2272 external_mini_cluster.cc:1428] Setting key 370fe988e1edf0e45535fd690ace7824
I20260928 12:25:20.390707 22901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:25:20.392333 22901 catalog_manager.cc:1347] Generated new cluster ID: 2080923ba126438580d53763f20968d0
I20260928 12:25:20.392383 22901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:25:20 dist-test-slave-920p krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 12:25:20.398452 22901 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:25:20.398908 22901 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:25:20.400985 22901 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26a23a4466f14f999d095bf4a41ea7cd: Generated new TSK 0
I20260928 12:25:20.401115 22901 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:25:20.402783 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:20.390543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46435 (local address 127.0.0.1:36683)
0928 12:25:20.390839 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:20.390848 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:20.390884 (+ 36us) server_negotiation.cc:407] Connection header received
0928 12:25:20.390998 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:20.391005 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:20.391072 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:20.391209 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:20.391807 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:20.393133 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:20.394186 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:20.394771 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:20.395141 (+ 370us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:20.397498 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:20.397502 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:20.397503 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:20.397514 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:20.401487 (+ 3973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:20.401843 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:20.401845 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:20.401845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:20.401918 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:20.402076 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:20.402076 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:20.402076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:20.402252 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:20.402339 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:20.402465 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 12:25:20.402565 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":81,"threads_started":1}
I20260928 12:25:20.403584 2272 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:25:20.410094 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:20.404772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55957 (local address 127.0.0.1:36683)
0928 12:25:20.404810 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:20.404811 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:20.404912 (+ 101us) server_negotiation.cc:407] Connection header received
0928 12:25:20.404964 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:20.404964 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:20.404975 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:20.405017 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:20.405510 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:20.405975 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:20.406648 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:20.406870 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:20.406921 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:20.407901 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:20.407903 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:20.407903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:20.407909 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:20.409069 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:20.409438 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:20.409439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:20.409439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:20.409490 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:20.409645 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:20.409646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:20.409646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:20.409750 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:20.409833 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:20.409937 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 12:25:20.410011 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:25:20.411759 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 22826
2026-09-28T12:25:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1208 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:25:20 dist-test-slave-920p krb5kdc[22925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:25:20 dist-test-slave-920p krb5kdc[22925](info): set up 2 sockets
Sep 28 12:25:20 dist-test-slave-920p krb5kdc[22925](info): commencing operation
krb5kdc: starting...
W20260928 12:25:21.349103 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.003s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:25:21 dist-test-slave-920p krb5kdc[22925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:25:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:25:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 12:25:21.458379 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44309
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44309
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:25:21.535709 22938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:21.535907 22938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:21.535935 22938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:21.537060 22938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:25:21.537106 22938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:21.537123 22938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:21.537139 22938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:25:21.537158 22938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:25:21.538643 22938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:21.538869 22938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:21.539063 22938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:21.543509 22944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:21.543635 22938 server_base.cc:1034] running on GCE node
W20260928 12:25:21.543511 22943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:21.543557 22946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:21.543964 22938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:21.544214 22938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:21.545343 22938 hybrid_clock.cc:648] HybridClock initialized: now 1790598321545325 us; error 36 us; skew 500 ppm
Sep 28 12:25:21 dist-test-slave-920p krb5kdc[22925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:21.554085 22938 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 12:25:21.554491 22938 webserver.cc:460] Webserver started at http://127.0.0.1:45087/ using document root <none> and password file <none>
I20260928 12:25:21.554685 22938 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:21.554733 22938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:21.554824 22938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:21.555843 22938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "c9b3c786649046fe890d1d578966139c"
format_stamp: "Formatted at 2026-09-28 12:25:21 on dist-test-slave-920p"
server_key: "a74239016746f89d04c59fcfe14bba72"
server_key_iv: "9c8837573d7305c3900171da8aec3323"
server_key_version: "encryptionkey@0"
I20260928 12:25:21.556141 22938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "c9b3c786649046fe890d1d578966139c"
format_stamp: "Formatted at 2026-09-28 12:25:21 on dist-test-slave-920p"
server_key: "a74239016746f89d04c59fcfe14bba72"
server_key_iv: "9c8837573d7305c3900171da8aec3323"
server_key_version: "encryptionkey@0"
I20260928 12:25:21.557372 22938 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:25:21.558069 22953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:21.558220 22938 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:25:21.558301 22938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "c9b3c786649046fe890d1d578966139c"
format_stamp: "Formatted at 2026-09-28 12:25:21 on dist-test-slave-920p"
server_key: "a74239016746f89d04c59fcfe14bba72"
server_key_iv: "9c8837573d7305c3900171da8aec3323"
server_key_version: "encryptionkey@0"
I20260928 12:25:21.558357 22938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:21.569761 22938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:21.571815 22938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:21.571921 22938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:21.575559 22938 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44309
I20260928 12:25:21.575611 23005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44309 every 8 connection(s)
I20260928 12:25:21.575905 22938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:25:21.576869 23006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:21.579329 23006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9b3c786649046fe890d1d578966139c: Bootstrap starting.
I20260928 12:25:21.579905 23006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9b3c786649046fe890d1d578966139c: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:21.580204 23006 log.cc:824] T 00000000000000000000000000000000 P c9b3c786649046fe890d1d578966139c: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:21.580883 23006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9b3c786649046fe890d1d578966139c: No bootstrap required, opened a new log
I20260928 12:25:21.582554 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 22938
I20260928 12:25:21.582679 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:25:21.582855 2272 external_mini_cluster.cc:1428] Setting key 8d68132b4d6cd2b72eefb5e5cb619058
I20260928 12:25:21.582922 23006 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9b3c786649046fe890d1d578966139c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9b3c786649046fe890d1d578966139c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44309 } }
I20260928 12:25:21.583089 23006 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9b3c786649046fe890d1d578966139c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:21.583115 23006 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9b3c786649046fe890d1d578966139c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9b3c786649046fe890d1d578966139c, State: Initialized, Role: FOLLOWER
I20260928 12:25:21.583250 23006 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9b3c786649046fe890d1d578966139c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9b3c786649046fe890d1d578966139c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44309 } }
I20260928 12:25:21.583310 23006 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9b3c786649046fe890d1d578966139c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:25:21.583338 23006 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9b3c786649046fe890d1d578966139c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:25:21.583401 23006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9b3c786649046fe890d1d578966139c [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:21.584090 23006 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9b3c786649046fe890d1d578966139c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9b3c786649046fe890d1d578966139c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44309 } }
I20260928 12:25:21.584195 23006 leader_election.cc:302] T 00000000000000000000000000000000 P c9b3c786649046fe890d1d578966139c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9b3c786649046fe890d1d578966139c; no voters:
I20260928 12:25:21.584352 23006 leader_election.cc:288] T 00000000000000000000000000000000 P c9b3c786649046fe890d1d578966139c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:25:21.584440 23011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9b3c786649046fe890d1d578966139c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:25:21.584599 23006 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9b3c786649046fe890d1d578966139c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:25:21.584661 23011 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9b3c786649046fe890d1d578966139c [term 1 LEADER]: Becoming Leader. State: Replica: c9b3c786649046fe890d1d578966139c, State: Running, Role: LEADER
I20260928 12:25:21.584811 23011 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9b3c786649046fe890d1d578966139c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9b3c786649046fe890d1d578966139c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44309 } }
I20260928 12:25:21.585152 23012 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9b3c786649046fe890d1d578966139c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9b3c786649046fe890d1d578966139c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9b3c786649046fe890d1d578966139c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44309 } } }
I20260928 12:25:21.585232 23012 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9b3c786649046fe890d1d578966139c [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:21.585469 23013 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9b3c786649046fe890d1d578966139c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9b3c786649046fe890d1d578966139c. Latest consensus state: current_term: 1 leader_uuid: "c9b3c786649046fe890d1d578966139c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9b3c786649046fe890d1d578966139c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44309 } } }
I20260928 12:25:21.585562 23013 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9b3c786649046fe890d1d578966139c [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:21.585883 23020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:25:21.586464 23020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:25:21.588809 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:21.583413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35969 (local address 127.0.0.1:44309)
0928 12:25:21.583594 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:21.583603 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:21.583637 (+ 34us) server_negotiation.cc:407] Connection header received
0928 12:25:21.584030 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:21.584039 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:21.584102 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:21.584259 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:21.584496 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:21.585851 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:21.586911 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:21.587496 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:21.587604 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:21.588203 (+ 599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:21.588206 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:21.588207 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:21.588373 (+ 166us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:21.588403 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:21.588413 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:21.588463 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:21.588559 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 12:25:21.588606 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":94,"threads_started":1}
I20260928 12:25:21.589672 23020 catalog_manager.cc:1347] Generated new cluster ID: 4f4408e94b5e45ba9d82a19cc9c26b92
I20260928 12:25:21.589725 23020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:25:21.589761 2272 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:25:21.601907 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:21.597624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44435 (local address 127.0.0.1:44309)
0928 12:25:21.597674 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:21.597675 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:21.597813 (+ 138us) server_negotiation.cc:407] Connection header received
0928 12:25:21.597877 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:21.597878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:21.597890 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:21.597931 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:21.598583 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:21.599188 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:21.600010 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:21.600265 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:21.600330 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:21.601581 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:21.601584 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:21.601584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:21.601601 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:21.601625 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:21.601631 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:21.601662 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:21.601766 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 12:25:21.601816 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 12:25:21.632889 23020 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:25:21.633677 23020 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:25:21.637980 23020 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9b3c786649046fe890d1d578966139c: Generated new TSK 0
I20260928 12:25:21.638171 23020 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:25:21.650604 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 22938
2026-09-28T12:25:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1244 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:25:21 dist-test-slave-920p krb5kdc[23037](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:25:21 dist-test-slave-920p krb5kdc[23037](info): set up 2 sockets
Sep 28 12:25:21 dist-test-slave-920p krb5kdc[23037](info): commencing operation
krb5kdc: starting...
W20260928 12:25:22.597168 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:25:22 dist-test-slave-920p krb5kdc[23037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:25:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:25:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 12:25:22.707369 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43157
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:25:22.785084 23050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:22.785306 23050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:22.785339 23050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:22.786458 23050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:25:22.786495 23050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:22.786513 23050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:22.786528 23050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:25:22.786553 23050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:25:22.787963 23050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:22.788195 23050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:22.788378 23050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:22.793042 23058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:22.793164 23056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:22.793208 23055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:22.793453 23050 server_base.cc:1034] running on GCE node
I20260928 12:25:22.793684 23050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:22.793938 23050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:22.795073 23050 hybrid_clock.cc:648] HybridClock initialized: now 1790598322795057 us; error 32 us; skew 500 ppm
Sep 28 12:25:22 dist-test-slave-920p krb5kdc[23037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:22.797379 23050 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 12:25:22.797798 23050 webserver.cc:460] Webserver started at http://127.0.0.1:41453/ using document root <none> and password file <none>
I20260928 12:25:22.798022 23050 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:22.798070 23050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:22.798192 23050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:22.799196 23050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "79ff48270a224ceca705b5b057ebfe04"
format_stamp: "Formatted at 2026-09-28 12:25:22 on dist-test-slave-920p"
server_key: "22f2dd8e2f05fccab3cf55947f0456eb"
server_key_iv: "2d03ac47bbfda5d669f7d369806d0c7e"
server_key_version: "encryptionkey@0"
I20260928 12:25:22.799522 23050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "79ff48270a224ceca705b5b057ebfe04"
format_stamp: "Formatted at 2026-09-28 12:25:22 on dist-test-slave-920p"
server_key: "22f2dd8e2f05fccab3cf55947f0456eb"
server_key_iv: "2d03ac47bbfda5d669f7d369806d0c7e"
server_key_version: "encryptionkey@0"
I20260928 12:25:22.800686 23050 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:25:22.801400 23065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:22.801532 23050 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:25:22.801611 23050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "79ff48270a224ceca705b5b057ebfe04"
format_stamp: "Formatted at 2026-09-28 12:25:22 on dist-test-slave-920p"
server_key: "22f2dd8e2f05fccab3cf55947f0456eb"
server_key_iv: "2d03ac47bbfda5d669f7d369806d0c7e"
server_key_version: "encryptionkey@0"
I20260928 12:25:22.801678 23050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:22.804787 23050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:22.806852 23050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:22.806980 23050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:22.810604 23050 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43157
I20260928 12:25:22.810662 23117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43157 every 8 connection(s)
I20260928 12:25:22.810971 23050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:25:22.811124 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 23050
I20260928 12:25:22.811229 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:25:22.811427 2272 external_mini_cluster.cc:1428] Setting key 08d8f7a4052fd6e099e57fbe552e7cc1
I20260928 12:25:22.811975 23118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:22.813614 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:22.811907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53287 (local address 127.0.0.1:43157)
0928 12:25:22.812075 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:22.812084 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:22.812114 (+ 30us) server_negotiation.cc:407] Connection header received
0928 12:25:22.812520 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:22.812526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:22.812549 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:22.812653 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:22.813202 (+ 549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:22.813205 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:22.813205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:22.813209 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:22.813242 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:22.813251 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:22.813328 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:22.813389 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 12:25:22.813426 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":64,"threads_started":1}
I20260928 12:25:22.814435 2272 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:25:22.814925 23118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79ff48270a224ceca705b5b057ebfe04: Bootstrap starting.
I20260928 12:25:22.815632 23118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79ff48270a224ceca705b5b057ebfe04: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:22.815934 23118 log.cc:824] T 00000000000000000000000000000000 P 79ff48270a224ceca705b5b057ebfe04: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:22.816793 23118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79ff48270a224ceca705b5b057ebfe04: No bootstrap required, opened a new log
I20260928 12:25:22.816901 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:22.815970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54517 (local address 127.0.0.1:43157)
0928 12:25:22.816009 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:22.816010 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:22.816117 (+ 107us) server_negotiation.cc:407] Connection header received
0928 12:25:22.816168 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:22.816169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:22.816175 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:22.816236 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:22.816717 (+ 481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:22.816718 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:22.816718 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:22.816721 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:22.816734 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:22.816738 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:22.816779 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:22.816832 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 12:25:22.816842 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:25:22.818608 23118 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79ff48270a224ceca705b5b057ebfe04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ff48270a224ceca705b5b057ebfe04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43157 } }
I20260928 12:25:22.818748 23118 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79ff48270a224ceca705b5b057ebfe04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:22.818769 23118 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79ff48270a224ceca705b5b057ebfe04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79ff48270a224ceca705b5b057ebfe04, State: Initialized, Role: FOLLOWER
I20260928 12:25:22.818866 23118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79ff48270a224ceca705b5b057ebfe04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ff48270a224ceca705b5b057ebfe04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43157 } }
I20260928 12:25:22.818943 23118 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79ff48270a224ceca705b5b057ebfe04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:25:22.818966 23118 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79ff48270a224ceca705b5b057ebfe04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:25:22.819011 23118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79ff48270a224ceca705b5b057ebfe04 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:22.819509 23118 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79ff48270a224ceca705b5b057ebfe04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ff48270a224ceca705b5b057ebfe04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43157 } }
I20260928 12:25:22.819604 23118 leader_election.cc:302] T 00000000000000000000000000000000 P 79ff48270a224ceca705b5b057ebfe04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79ff48270a224ceca705b5b057ebfe04; no voters:
I20260928 12:25:22.819761 23118 leader_election.cc:288] T 00000000000000000000000000000000 P 79ff48270a224ceca705b5b057ebfe04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:25:22.819869 23128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79ff48270a224ceca705b5b057ebfe04 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:25:22.819995 23118 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79ff48270a224ceca705b5b057ebfe04 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:25:22.820057 23128 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79ff48270a224ceca705b5b057ebfe04 [term 1 LEADER]: Becoming Leader. State: Replica: 79ff48270a224ceca705b5b057ebfe04, State: Running, Role: LEADER
I20260928 12:25:22.820149 23128 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79ff48270a224ceca705b5b057ebfe04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ff48270a224ceca705b5b057ebfe04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43157 } }
I20260928 12:25:22.820478 23129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79ff48270a224ceca705b5b057ebfe04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79ff48270a224ceca705b5b057ebfe04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ff48270a224ceca705b5b057ebfe04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43157 } } }
I20260928 12:25:22.820477 23130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79ff48270a224ceca705b5b057ebfe04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79ff48270a224ceca705b5b057ebfe04. Latest consensus state: current_term: 1 leader_uuid: "79ff48270a224ceca705b5b057ebfe04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ff48270a224ceca705b5b057ebfe04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43157 } } }
I20260928 12:25:22.820616 23129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79ff48270a224ceca705b5b057ebfe04 [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:22.820623 23130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79ff48270a224ceca705b5b057ebfe04 [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:22.820806 23137 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:25:22.821331 23137 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:25:22.822770 23137 catalog_manager.cc:1347] Generated new cluster ID: 9cea9a6426264df9afebc35499affbff
I20260928 12:25:22.822820 23137 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:25:22.833232 23137 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:25:22.833400 23137 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:25:22.836941 23137 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79ff48270a224ceca705b5b057ebfe04: Generated new TSK 0
I20260928 12:25:22.837092 23137 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:25:22.840688 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 23050
2026-09-28T12:25:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1184 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:25:22 dist-test-slave-920p krb5kdc[23149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:25:22 dist-test-slave-920p krb5kdc[23149](info): set up 2 sockets
Sep 28 12:25:22 dist-test-slave-920p krb5kdc[23149](info): commencing operation
krb5kdc: starting...
W20260928 12:25:23.785035 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:25:23 dist-test-slave-920p krb5kdc[23149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:25:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:25:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 12:25:23.896962 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45241
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:25:23.976974 23162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:23.977183 23162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:23.977205 23162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:23.978502 23162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:25:23.978555 23162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:23.978569 23162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:23.978580 23162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:25:23.978591 23162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:25:23.980142 23162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:23.980341 23162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:23.980511 23162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:23.985216 23167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:23.985301 23162 server_base.cc:1034] running on GCE node
W20260928 12:25:23.985220 23170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:23.985260 23168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:23.985656 23162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:23.985948 23162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:23.987126 23162 hybrid_clock.cc:648] HybridClock initialized: now 1790598323987106 us; error 37 us; skew 500 ppm
Sep 28 12:25:23 dist-test-slave-920p krb5kdc[23149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:23.994515 23162 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 12:25:23.994922 23162 webserver.cc:460] Webserver started at http://127.0.0.1:44691/ using document root <none> and password file <none>
I20260928 12:25:23.995106 23162 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:23.995147 23162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:23.995235 23162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:23.996219 23162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "22d424c9ea2f4ce18f28778932937998"
format_stamp: "Formatted at 2026-09-28 12:25:23 on dist-test-slave-920p"
server_key: "cb65dbb70980aca8700efa9847bbec88"
server_key_iv: "48a38bcc7dc36c7b6a1de111a39683b4"
server_key_version: "encryptionkey@0"
I20260928 12:25:23.996500 23162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "22d424c9ea2f4ce18f28778932937998"
format_stamp: "Formatted at 2026-09-28 12:25:23 on dist-test-slave-920p"
server_key: "cb65dbb70980aca8700efa9847bbec88"
server_key_iv: "48a38bcc7dc36c7b6a1de111a39683b4"
server_key_version: "encryptionkey@0"
I20260928 12:25:23.997656 23162 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 12:25:23.998266 23177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:23.998420 23162 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:25:23.998498 23162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "22d424c9ea2f4ce18f28778932937998"
format_stamp: "Formatted at 2026-09-28 12:25:23 on dist-test-slave-920p"
server_key: "cb65dbb70980aca8700efa9847bbec88"
server_key_iv: "48a38bcc7dc36c7b6a1de111a39683b4"
server_key_version: "encryptionkey@0"
I20260928 12:25:23.998551 23162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:24.010174 23162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:24.011745 23162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:24.011839 23162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:24.015347 23162 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45241
I20260928 12:25:24.015375 23229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45241 every 8 connection(s)
I20260928 12:25:24.015672 23162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:25:24.016554 23230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:24.018715 23230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22d424c9ea2f4ce18f28778932937998: Bootstrap starting.
I20260928 12:25:24.019244 23230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22d424c9ea2f4ce18f28778932937998: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:24.019486 23230 log.cc:824] T 00000000000000000000000000000000 P 22d424c9ea2f4ce18f28778932937998: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:24.020184 23230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22d424c9ea2f4ce18f28778932937998: No bootstrap required, opened a new log
I20260928 12:25:24.021416 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 23162
I20260928 12:25:24.021514 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:25:24.021675 2272 external_mini_cluster.cc:1428] Setting key e14ff19d23aa86825a24d0b26d91c6a2
I20260928 12:25:24.022070 23230 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22d424c9ea2f4ce18f28778932937998 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22d424c9ea2f4ce18f28778932937998" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45241 } }
I20260928 12:25:24.022305 23230 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22d424c9ea2f4ce18f28778932937998 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:24.022360 23230 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22d424c9ea2f4ce18f28778932937998 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22d424c9ea2f4ce18f28778932937998, State: Initialized, Role: FOLLOWER
I20260928 12:25:24.022473 23230 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22d424c9ea2f4ce18f28778932937998 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22d424c9ea2f4ce18f28778932937998" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45241 } }
I20260928 12:25:24.022559 23230 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22d424c9ea2f4ce18f28778932937998 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:25:24.022596 23230 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22d424c9ea2f4ce18f28778932937998 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:25:24.022643 23230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22d424c9ea2f4ce18f28778932937998 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:24.023664 23230 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22d424c9ea2f4ce18f28778932937998 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22d424c9ea2f4ce18f28778932937998" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45241 } }
I20260928 12:25:24.023752 23230 leader_election.cc:302] T 00000000000000000000000000000000 P 22d424c9ea2f4ce18f28778932937998 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22d424c9ea2f4ce18f28778932937998; no voters:
I20260928 12:25:24.023896 23230 leader_election.cc:288] T 00000000000000000000000000000000 P 22d424c9ea2f4ce18f28778932937998 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:25:24.023926 23235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22d424c9ea2f4ce18f28778932937998 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:25:24.024081 23235 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22d424c9ea2f4ce18f28778932937998 [term 1 LEADER]: Becoming Leader. State: Replica: 22d424c9ea2f4ce18f28778932937998, State: Running, Role: LEADER
I20260928 12:25:24.024143 23230 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22d424c9ea2f4ce18f28778932937998 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:25:24.024209 23235 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22d424c9ea2f4ce18f28778932937998 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22d424c9ea2f4ce18f28778932937998" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45241 } }
I20260928 12:25:24.024540 23235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22d424c9ea2f4ce18f28778932937998 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22d424c9ea2f4ce18f28778932937998. Latest consensus state: current_term: 1 leader_uuid: "22d424c9ea2f4ce18f28778932937998" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22d424c9ea2f4ce18f28778932937998" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45241 } } }
I20260928 12:25:24.024770 23235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22d424c9ea2f4ce18f28778932937998 [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:24.024693 23237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22d424c9ea2f4ce18f28778932937998 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22d424c9ea2f4ce18f28778932937998" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22d424c9ea2f4ce18f28778932937998" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45241 } } }
I20260928 12:25:24.025195 23237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22d424c9ea2f4ce18f28778932937998 [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:24.025208 23244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:25:24.025796 23244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:25:24.027333 23244 catalog_manager.cc:1347] Generated new cluster ID: 5c6e5f8fa8f54283ab945fa4ce4fd14e
I20260928 12:25:24.027383 23244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:25:24 dist-test-slave-920p krb5kdc[23149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 12:25:24.032457 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:24.022126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.208:39581 (local address 127.0.0.1:45241)
0928 12:25:24.022300 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:24.022309 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:24.022341 (+ 32us) server_negotiation.cc:407] Connection header received
0928 12:25:24.022447 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:24.022454 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:24.022509 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:24.022655 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:24.023510 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:24.024709 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:24.025766 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:24.026352 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:24.026452 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:24.029122 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:24.029125 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:24.029126 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:24.029136 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:24.030929 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:24.031338 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:24.031340 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:24.031341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:24.031439 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:24.031661 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:24.031661 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:24.031662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:24.031917 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:24.032059 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:24.032181 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 12:25:24.032275 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
I20260928 12:25:24.033542 2272 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:25:24.048414 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:24.036655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.208:42311 (local address 127.0.0.1:45241)
0928 12:25:24.036687 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:24.036689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:24.041417 (+ 4728us) server_negotiation.cc:407] Connection header received
0928 12:25:24.041456 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:24.041457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:24.041468 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:24.041532 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:24.042099 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:24.042893 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:24.043827 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:24.044394 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:24.044469 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:24.045794 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:24.045795 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:24.045796 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:24.045802 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:24.047316 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:24.047669 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:24.047670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:24.047670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:24.047726 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:24.047913 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:24.047913 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:24.047914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:24.048054 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:24.048162 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:24.048253 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 12:25:24.048316 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 12:25:24.050803 23244 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:25:24.051514 23244 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:25:24.054943 23244 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22d424c9ea2f4ce18f28778932937998: Generated new TSK 0
I20260928 12:25:24.055102 23244 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:25:24.060424 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 23162
W20260928 12:25:24.064083 23147 connection.cc:535] client connection to 127.0.0.1:45241 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45241): Connection reset by peer (error 104)
2026-09-28T12:25:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1219 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:25:24 dist-test-slave-920p krb5kdc[23261](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:25:24 dist-test-slave-920p krb5kdc[23261](info): set up 2 sockets
Sep 28 12:25:24 dist-test-slave-920p krb5kdc[23261](info): commencing operation
krb5kdc: starting...
W20260928 12:25:25.021212 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:25:25 dist-test-slave-920p krb5kdc[23261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:25:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:25:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 12:25:25.133180 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35223
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:25:25.210450 23274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:25.210649 23274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:25.210670 23274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:25.211786 23274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:25:25.211825 23274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:25.211838 23274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:25.211849 23274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:25:25.211859 23274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:25:25.213311 23274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:25.213533 23274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:25.213732 23274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:25.218240 23279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:25.218245 23280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:25.218362 23274 server_base.cc:1034] running on GCE node
W20260928 12:25:25.218250 23282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:25.218653 23274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:25.218889 23274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:25.220048 23274 hybrid_clock.cc:648] HybridClock initialized: now 1790598325220032 us; error 27 us; skew 500 ppm
Sep 28 12:25:25 dist-test-slave-920p krb5kdc[23261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:25.222568 23274 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 12:25:25.222983 23274 webserver.cc:460] Webserver started at http://127.0.0.1:40441/ using document root <none> and password file <none>
I20260928 12:25:25.223183 23274 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:25.223258 23274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:25.223389 23274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:25.224452 23274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "5ecaa0c2293041acb9b25b1ddb1fe2ed"
format_stamp: "Formatted at 2026-09-28 12:25:25 on dist-test-slave-920p"
server_key: "5c98ef05446bde3aba93b6213f5d0dfe"
server_key_iv: "d88dad4b040ee007793f0d9b947ce4c7"
server_key_version: "encryptionkey@0"
I20260928 12:25:25.224781 23274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "5ecaa0c2293041acb9b25b1ddb1fe2ed"
format_stamp: "Formatted at 2026-09-28 12:25:25 on dist-test-slave-920p"
server_key: "5c98ef05446bde3aba93b6213f5d0dfe"
server_key_iv: "d88dad4b040ee007793f0d9b947ce4c7"
server_key_version: "encryptionkey@0"
I20260928 12:25:25.226095 23274 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:25:25.226845 23289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:25.227026 23274 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:25:25.227101 23274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "5ecaa0c2293041acb9b25b1ddb1fe2ed"
format_stamp: "Formatted at 2026-09-28 12:25:25 on dist-test-slave-920p"
server_key: "5c98ef05446bde3aba93b6213f5d0dfe"
server_key_iv: "d88dad4b040ee007793f0d9b947ce4c7"
server_key_version: "encryptionkey@0"
I20260928 12:25:25.227169 23274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:25.246510 23274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:25.248255 23274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:25.248368 23274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:25.251981 23274 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35223
I20260928 12:25:25.252025 23341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35223 every 8 connection(s)
I20260928 12:25:25.252321 23274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:25:25.253228 23342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:25.255362 23342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ecaa0c2293041acb9b25b1ddb1fe2ed: Bootstrap starting.
I20260928 12:25:25.255887 23342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ecaa0c2293041acb9b25b1ddb1fe2ed: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:25.256146 23342 log.cc:824] T 00000000000000000000000000000000 P 5ecaa0c2293041acb9b25b1ddb1fe2ed: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:25.256752 23342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ecaa0c2293041acb9b25b1ddb1fe2ed: No bootstrap required, opened a new log
I20260928 12:25:25.257491 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 23274
I20260928 12:25:25.257612 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:25:25.257784 2272 external_mini_cluster.cc:1428] Setting key 76b2c52f6e41f41090b99c0b157727d4
I20260928 12:25:25.258612 23342 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ecaa0c2293041acb9b25b1ddb1fe2ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ecaa0c2293041acb9b25b1ddb1fe2ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35223 } }
I20260928 12:25:25.258769 23342 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ecaa0c2293041acb9b25b1ddb1fe2ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:25.258802 23342 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ecaa0c2293041acb9b25b1ddb1fe2ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ecaa0c2293041acb9b25b1ddb1fe2ed, State: Initialized, Role: FOLLOWER
I20260928 12:25:25.258932 23342 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ecaa0c2293041acb9b25b1ddb1fe2ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ecaa0c2293041acb9b25b1ddb1fe2ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35223 } }
I20260928 12:25:25.259006 23342 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ecaa0c2293041acb9b25b1ddb1fe2ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:25:25.259034 23342 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ecaa0c2293041acb9b25b1ddb1fe2ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:25:25.259079 23342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ecaa0c2293041acb9b25b1ddb1fe2ed [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:25.259546 23342 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ecaa0c2293041acb9b25b1ddb1fe2ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ecaa0c2293041acb9b25b1ddb1fe2ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35223 } }
I20260928 12:25:25.259634 23342 leader_election.cc:302] T 00000000000000000000000000000000 P 5ecaa0c2293041acb9b25b1ddb1fe2ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ecaa0c2293041acb9b25b1ddb1fe2ed; no voters:
I20260928 12:25:25.259781 23342 leader_election.cc:288] T 00000000000000000000000000000000 P 5ecaa0c2293041acb9b25b1ddb1fe2ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:25:25.259819 23347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ecaa0c2293041acb9b25b1ddb1fe2ed [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:25:25.259958 23347 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ecaa0c2293041acb9b25b1ddb1fe2ed [term 1 LEADER]: Becoming Leader. State: Replica: 5ecaa0c2293041acb9b25b1ddb1fe2ed, State: Running, Role: LEADER
I20260928 12:25:25.260020 23342 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ecaa0c2293041acb9b25b1ddb1fe2ed [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:25:25.260113 23347 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ecaa0c2293041acb9b25b1ddb1fe2ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ecaa0c2293041acb9b25b1ddb1fe2ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35223 } }
I20260928 12:25:25.260434 23347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ecaa0c2293041acb9b25b1ddb1fe2ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ecaa0c2293041acb9b25b1ddb1fe2ed. Latest consensus state: current_term: 1 leader_uuid: "5ecaa0c2293041acb9b25b1ddb1fe2ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ecaa0c2293041acb9b25b1ddb1fe2ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35223 } } }
I20260928 12:25:25.260637 23347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ecaa0c2293041acb9b25b1ddb1fe2ed [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:25.260577 23349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ecaa0c2293041acb9b25b1ddb1fe2ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ecaa0c2293041acb9b25b1ddb1fe2ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ecaa0c2293041acb9b25b1ddb1fe2ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35223 } } }
I20260928 12:25:25.260993 23349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ecaa0c2293041acb9b25b1ddb1fe2ed [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:25.261158 23356 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:25:25.261703 23356 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:25:25.263080 23356 catalog_manager.cc:1347] Generated new cluster ID: 2ff43f5315b84805818df74cf7993cc1
I20260928 12:25:25.263124 23356 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:25:25.263538 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:25.258302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.208:57303 (local address 127.0.0.1:35223)
0928 12:25:25.258478 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:25.258485 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:25.258512 (+ 27us) server_negotiation.cc:407] Connection header received
0928 12:25:25.258603 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:25.258608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:25.258650 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:25.258759 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:25.258953 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:25.260412 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:25.261509 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:25.262072 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:25.262185 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:25.262834 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:25.262837 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:25.262838 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:25.263060 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:25.263097 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:25.263112 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:25.263185 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:25.263320 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 12:25:25.263345 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":75,"threads_started":1}
I20260928 12:25:25.266137 2272 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:25:25.272588 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:25.267483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.208:48023 (local address 127.0.0.1:35223)
0928 12:25:25.267513 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:25.267515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:25.269418 (+ 1903us) server_negotiation.cc:407] Connection header received
0928 12:25:25.269505 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:25.269506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:25.269517 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:25.269572 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:25.269686 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:25.270435 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:25.271359 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:25.271676 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:25.271743 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:25.272363 (+ 620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:25.272364 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:25.272365 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:25.272369 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:25.272384 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:25.272388 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:25.272445 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:25.272500 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 12:25:25.272512 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 12:25:25.281749 23356 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:25:25.282295 23356 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:25:25.289851 23356 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ecaa0c2293041acb9b25b1ddb1fe2ed: Generated new TSK 0
I20260928 12:25:25.290053 23356 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:25:25.293555 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 23274
W20260928 12:25:25.297142 23259 connection.cc:535] client connection to 127.0.0.1:35223 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35223): Connection reset by peer (error 104)
2026-09-28T12:25:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1233 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:25:25 dist-test-slave-920p krb5kdc[23373](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:25:25 dist-test-slave-920p krb5kdc[23373](info): set up 2 sockets
Sep 28 12:25:25 dist-test-slave-920p krb5kdc[23373](info): commencing operation
krb5kdc: starting...
W20260928 12:25:26.249205 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:25:26 dist-test-slave-920p krb5kdc[23373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:25:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:25:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 12:25:26.372167 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42765
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:25:26.449849 23386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:26.450057 23386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:26.450079 23386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:26.451172 23386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:25:26.451206 23386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:26.451220 23386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:26.451231 23386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:25:26.451244 23386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:25:26.452621 23386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:26.452800 23386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:26.452960 23386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:26.457598 23391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:26.457589 23392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:26.457589 23394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:26.457604 23386 server_base.cc:1034] running on GCE node
I20260928 12:25:26.458002 23386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:26.458317 23386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:26.459467 23386 hybrid_clock.cc:648] HybridClock initialized: now 1790598326459454 us; error 31 us; skew 500 ppm
Sep 28 12:25:26 dist-test-slave-920p krb5kdc[23373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:26.461728 23386 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 12:25:26.462109 23386 webserver.cc:460] Webserver started at http://127.0.0.1:42493/ using document root <none> and password file <none>
I20260928 12:25:26.462307 23386 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:26.462349 23386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:26.462466 23386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:26.463470 23386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "24a4c5c4dfda48faa82f0c72a132e3ae"
format_stamp: "Formatted at 2026-09-28 12:25:26 on dist-test-slave-920p"
server_key: "74a62e29262f9f4087b0ae005a0b6e21"
server_key_iv: "4b96d0e77ea3648bbb6309277082b286"
server_key_version: "encryptionkey@0"
I20260928 12:25:26.463783 23386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "24a4c5c4dfda48faa82f0c72a132e3ae"
format_stamp: "Formatted at 2026-09-28 12:25:26 on dist-test-slave-920p"
server_key: "74a62e29262f9f4087b0ae005a0b6e21"
server_key_iv: "4b96d0e77ea3648bbb6309277082b286"
server_key_version: "encryptionkey@0"
I20260928 12:25:26.464927 23386 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:25:26.465567 23401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:26.465693 23386 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:26.465772 23386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "24a4c5c4dfda48faa82f0c72a132e3ae"
format_stamp: "Formatted at 2026-09-28 12:25:26 on dist-test-slave-920p"
server_key: "74a62e29262f9f4087b0ae005a0b6e21"
server_key_iv: "4b96d0e77ea3648bbb6309277082b286"
server_key_version: "encryptionkey@0"
I20260928 12:25:26.465831 23386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:26.468797 23386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:26.470417 23386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:26.470532 23386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:26.474044 23386 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42765
I20260928 12:25:26.474073 23453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42765 every 8 connection(s)
I20260928 12:25:26.474352 23386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:25:26.475231 23454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:26.476357 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 23386
I20260928 12:25:26.476454 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:25:26.476605 2272 external_mini_cluster.cc:1428] Setting key 5e8c04030c05b56aad9a842a7021440b
I20260928 12:25:26.477519 23454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24a4c5c4dfda48faa82f0c72a132e3ae: Bootstrap starting.
I20260928 12:25:26.478101 23454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24a4c5c4dfda48faa82f0c72a132e3ae: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:26.478350 23454 log.cc:824] T 00000000000000000000000000000000 P 24a4c5c4dfda48faa82f0c72a132e3ae: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:26.478729 23457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:26.477051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.208:35411 (local address 127.0.0.1:42765)
0928 12:25:26.477205 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:26.477212 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:26.477234 (+ 22us) server_negotiation.cc:407] Connection header received
0928 12:25:26.477689 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:26.477695 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:26.477718 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:26.477826 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:26.478391 (+ 565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:26.478394 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:26.478394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:26.478398 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:26.478432 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:26.478446 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:26.478520 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:26.478550 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 12:25:26.478589 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":57,"threads_started":1}
I20260928 12:25:26.478987 23454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24a4c5c4dfda48faa82f0c72a132e3ae: No bootstrap required, opened a new log
I20260928 12:25:26.479518 2272 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:25:26.480876 23454 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24a4c5c4dfda48faa82f0c72a132e3ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a4c5c4dfda48faa82f0c72a132e3ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42765 } }
I20260928 12:25:26.481001 23454 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24a4c5c4dfda48faa82f0c72a132e3ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:26.481031 23454 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24a4c5c4dfda48faa82f0c72a132e3ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a4c5c4dfda48faa82f0c72a132e3ae, State: Initialized, Role: FOLLOWER
I20260928 12:25:26.481137 23454 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24a4c5c4dfda48faa82f0c72a132e3ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a4c5c4dfda48faa82f0c72a132e3ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42765 } }
I20260928 12:25:26.481204 23454 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24a4c5c4dfda48faa82f0c72a132e3ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:25:26.481232 23454 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24a4c5c4dfda48faa82f0c72a132e3ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:25:26.481310 23454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24a4c5c4dfda48faa82f0c72a132e3ae [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:26.481946 23454 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24a4c5c4dfda48faa82f0c72a132e3ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a4c5c4dfda48faa82f0c72a132e3ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42765 } }
I20260928 12:25:26.482050 23454 leader_election.cc:302] T 00000000000000000000000000000000 P 24a4c5c4dfda48faa82f0c72a132e3ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24a4c5c4dfda48faa82f0c72a132e3ae; no voters:
I20260928 12:25:26.482200 23454 leader_election.cc:288] T 00000000000000000000000000000000 P 24a4c5c4dfda48faa82f0c72a132e3ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:25:26.482232 23464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24a4c5c4dfda48faa82f0c72a132e3ae [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:25:26.482373 23464 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24a4c5c4dfda48faa82f0c72a132e3ae [term 1 LEADER]: Becoming Leader. State: Replica: 24a4c5c4dfda48faa82f0c72a132e3ae, State: Running, Role: LEADER
I20260928 12:25:26.482429 23457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:26.481472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.208:52709 (local address 127.0.0.1:42765)
0928 12:25:26.481520 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:26.481522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:26.481609 (+ 87us) server_negotiation.cc:407] Connection header received
0928 12:25:26.481670 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:26.481671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:26.481674 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:26.481714 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:26.482142 (+ 428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:26.482144 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:26.482144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:26.482148 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:26.482160 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:26.482164 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:26.482213 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:26.482375 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 12:25:26.482386 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 12:25:26.482432 23454 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24a4c5c4dfda48faa82f0c72a132e3ae [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:25:26.482483 23464 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24a4c5c4dfda48faa82f0c72a132e3ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a4c5c4dfda48faa82f0c72a132e3ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42765 } }
I20260928 12:25:26.482908 23464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24a4c5c4dfda48faa82f0c72a132e3ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24a4c5c4dfda48faa82f0c72a132e3ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a4c5c4dfda48faa82f0c72a132e3ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42765 } } }
I20260928 12:25:26.482985 23464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24a4c5c4dfda48faa82f0c72a132e3ae [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:26.483105 23464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24a4c5c4dfda48faa82f0c72a132e3ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24a4c5c4dfda48faa82f0c72a132e3ae. Latest consensus state: current_term: 1 leader_uuid: "24a4c5c4dfda48faa82f0c72a132e3ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a4c5c4dfda48faa82f0c72a132e3ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42765 } } }
I20260928 12:25:26.483170 23473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:25:26.483198 23464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24a4c5c4dfda48faa82f0c72a132e3ae [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:26.483603 23473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:25:26.485103 23473 catalog_manager.cc:1347] Generated new cluster ID: 891e52583a2046eaae6e80807f026287
I20260928 12:25:26.485146 23473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:25:26.504196 23473 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:25:26.504360 23473 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:25:26.511830 23473 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24a4c5c4dfda48faa82f0c72a132e3ae: Generated new TSK 0
I20260928 12:25:26.511991 23473 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:25:26.518462 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 23386
2026-09-28T12:25:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1225 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:25:26 dist-test-slave-920p krb5kdc[23485](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:25:26 dist-test-slave-920p krb5kdc[23485](info): set up 2 sockets
Sep 28 12:25:26 dist-test-slave-920p krb5kdc[23485](info): commencing operation
krb5kdc: starting...
W20260928 12:25:27.465157 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:25:27 dist-test-slave-920p krb5kdc[23485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:25:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:25:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 12:25:27.574250 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46661
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:25:27.652312 23498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:27.652529 23498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:27.652549 23498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:27.653671 23498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:25:27.653705 23498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:27.653718 23498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:27.653729 23498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:25:27.653740 23498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:25:27.655121 23498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46661
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:27.655299 23498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:27.655452 23498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:27.659940 23503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:27.659960 23504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:27.660099 23506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:27.660074 23498 server_base.cc:1034] running on GCE node
I20260928 12:25:27.660391 23498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:27.660645 23498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:27.661778 23498 hybrid_clock.cc:648] HybridClock initialized: now 1790598327661766 us; error 28 us; skew 500 ppm
Sep 28 12:25:27 dist-test-slave-920p krb5kdc[23485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:27.663931 23498 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 12:25:27.664299 23498 webserver.cc:460] Webserver started at http://127.0.0.1:44503/ using document root <none> and password file <none>
I20260928 12:25:27.664481 23498 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:27.664526 23498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:27.664642 23498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:27.665652 23498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "f6022d9443074994b8a31e7073dc5b0a"
format_stamp: "Formatted at 2026-09-28 12:25:27 on dist-test-slave-920p"
server_key: "fbd39ee547e693ee8f0778c7e18218f0"
server_key_iv: "9aab00aa5be7898287065b3602c137ad"
server_key_version: "encryptionkey@0"
I20260928 12:25:27.665977 23498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "f6022d9443074994b8a31e7073dc5b0a"
format_stamp: "Formatted at 2026-09-28 12:25:27 on dist-test-slave-920p"
server_key: "fbd39ee547e693ee8f0778c7e18218f0"
server_key_iv: "9aab00aa5be7898287065b3602c137ad"
server_key_version: "encryptionkey@0"
I20260928 12:25:27.667116 23498 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:25:27.667762 23513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:27.667917 23498 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:25:27.667995 23498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "f6022d9443074994b8a31e7073dc5b0a"
format_stamp: "Formatted at 2026-09-28 12:25:27 on dist-test-slave-920p"
server_key: "fbd39ee547e693ee8f0778c7e18218f0"
server_key_iv: "9aab00aa5be7898287065b3602c137ad"
server_key_version: "encryptionkey@0"
I20260928 12:25:27.668056 23498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:27.681823 23498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:27.683636 23498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:27.683768 23498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:27.687476 23498 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46661
I20260928 12:25:27.687511 23565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46661 every 8 connection(s)
I20260928 12:25:27.687829 23498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:25:27.688082 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 23498
I20260928 12:25:27.688201 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:25:27.688403 2272 external_mini_cluster.cc:1428] Setting key d1f9b4cf6dccb9c4a52d52edcba832da
I20260928 12:25:27.688930 23566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:27.691744 23566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6022d9443074994b8a31e7073dc5b0a: Bootstrap starting.
I20260928 12:25:27.692428 23566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6022d9443074994b8a31e7073dc5b0a: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:27.692687 23566 log.cc:824] T 00000000000000000000000000000000 P f6022d9443074994b8a31e7073dc5b0a: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:27.693368 23566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6022d9443074994b8a31e7073dc5b0a: No bootstrap required, opened a new log
Sep 28 12:25:27 dist-test-slave-920p krb5kdc[23485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 12:25:27.695138 23566 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6022d9443074994b8a31e7073dc5b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6022d9443074994b8a31e7073dc5b0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46661 } }
I20260928 12:25:27.695308 23566 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6022d9443074994b8a31e7073dc5b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:27.695344 23566 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6022d9443074994b8a31e7073dc5b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6022d9443074994b8a31e7073dc5b0a, State: Initialized, Role: FOLLOWER
I20260928 12:25:27.695468 23566 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6022d9443074994b8a31e7073dc5b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6022d9443074994b8a31e7073dc5b0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46661 } }
I20260928 12:25:27.695557 23566 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6022d9443074994b8a31e7073dc5b0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:25:27.695595 23566 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6022d9443074994b8a31e7073dc5b0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:25:27.695644 23566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6022d9443074994b8a31e7073dc5b0a [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:27.696274 23566 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6022d9443074994b8a31e7073dc5b0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6022d9443074994b8a31e7073dc5b0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46661 } }
I20260928 12:25:27.696386 23566 leader_election.cc:302] T 00000000000000000000000000000000 P f6022d9443074994b8a31e7073dc5b0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6022d9443074994b8a31e7073dc5b0a; no voters:
I20260928 12:25:27.696549 23566 leader_election.cc:288] T 00000000000000000000000000000000 P f6022d9443074994b8a31e7073dc5b0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:25:27.696597 23571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6022d9443074994b8a31e7073dc5b0a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:25:27.696725 23571 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6022d9443074994b8a31e7073dc5b0a [term 1 LEADER]: Becoming Leader. State: Replica: f6022d9443074994b8a31e7073dc5b0a, State: Running, Role: LEADER
I20260928 12:25:27.696792 23566 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6022d9443074994b8a31e7073dc5b0a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:25:27.696841 23571 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6022d9443074994b8a31e7073dc5b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6022d9443074994b8a31e7073dc5b0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46661 } }
I20260928 12:25:27.697252 23572 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6022d9443074994b8a31e7073dc5b0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6022d9443074994b8a31e7073dc5b0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6022d9443074994b8a31e7073dc5b0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46661 } } }
I20260928 12:25:27.697350 23572 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6022d9443074994b8a31e7073dc5b0a [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:27.697283 23573 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6022d9443074994b8a31e7073dc5b0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6022d9443074994b8a31e7073dc5b0a. Latest consensus state: current_term: 1 leader_uuid: "f6022d9443074994b8a31e7073dc5b0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6022d9443074994b8a31e7073dc5b0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46661 } } }
I20260928 12:25:27.697413 23573 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6022d9443074994b8a31e7073dc5b0a [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:27.697880 23575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:25:27.698381 23575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:25:27.699039 23569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:27.688892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.208:60803 (local address 127.0.0.1:46661)
0928 12:25:27.689075 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:27.689084 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:27.689110 (+ 26us) server_negotiation.cc:407] Connection header received
0928 12:25:27.689572 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:27.689579 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:27.689625 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:27.689734 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:27.690502 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:27.691417 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:27.692446 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:27.692885 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:27.692970 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:27.695052 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:27.695055 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:27.695056 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:27.695067 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:27.696872 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:27.697375 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:27.697378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:27.697378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:27.697481 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:27.697743 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:27.697744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:27.697744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:27.698052 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:27.698435 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:27.698582 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 12:25:27.698795 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":62,"threads_started":1}
I20260928 12:25:27.699818 23575 catalog_manager.cc:1347] Generated new cluster ID: 456243e770dd48efa809c298224bca0b
I20260928 12:25:27.699844 2272 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:25:27.699867 23575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:25:27.706995 23575 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:25:27.707641 23575 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:25:27.710700 23569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:27.701066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.208:46529 (local address 127.0.0.1:46661)
0928 12:25:27.701099 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:27.701100 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:27.705456 (+ 4356us) server_negotiation.cc:407] Connection header received
0928 12:25:27.705502 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:27.705503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:27.705513 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:27.705553 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:27.706052 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:27.706520 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:27.707273 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:27.707489 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:27.707539 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:27.708543 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:27.708544 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:27.708545 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:27.708551 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:27.709728 (+ 1177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:27.710048 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:27.710049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:27.710049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:27.710098 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:27.710276 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:27.710277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:27.710277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:27.710373 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:27.710454 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:27.710568 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 12:25:27.710622 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 12:25:27.720386 23575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6022d9443074994b8a31e7073dc5b0a: Generated new TSK 0
I20260928 12:25:27.720566 23575 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:25:27.728340 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 23498
W20260928 12:25:27.731956 23483 connection.cc:535] client connection to 127.0.0.1:46661 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46661): Connection reset by peer (error 104)
2026-09-28T12:25:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1209 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:25:27 dist-test-slave-920p krb5kdc[23597](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:25:27 dist-test-slave-920p krb5kdc[23597](info): set up 2 sockets
Sep 28 12:25:27 dist-test-slave-920p krb5kdc[23597](info): commencing operation
krb5kdc: starting...
W20260928 12:25:28.669257 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:25:28 dist-test-slave-920p krb5kdc[23597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:25:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:25:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 12:25:28.779247 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43263
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:25:28.856547 23610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:28.856741 23610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:28.856762 23610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:28.858091 23610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:25:28.858147 23610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:28.858162 23610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:28.858173 23610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:25:28.858184 23610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:25:28.859754 23610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43263
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:28.859972 23610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:28.860183 23610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:28.864748 23618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:28.864846 23610 server_base.cc:1034] running on GCE node
W20260928 12:25:28.864732 23616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:28.864734 23615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:28.865195 23610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:28.865504 23610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:28.866640 23610 hybrid_clock.cc:648] HybridClock initialized: now 1790598328866625 us; error 34 us; skew 500 ppm
Sep 28 12:25:28 dist-test-slave-920p krb5kdc[23597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:28.869088 23610 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 12:25:28.869499 23610 webserver.cc:460] Webserver started at http://127.0.0.1:43795/ using document root <none> and password file <none>
I20260928 12:25:28.869689 23610 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:28.869728 23610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:28.869813 23610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:28.870786 23610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "241bb66827904aaa990979165d6a7a4e"
format_stamp: "Formatted at 2026-09-28 12:25:28 on dist-test-slave-920p"
server_key: "91bc7f0b81f7669c743db391ec260da3"
server_key_iv: "36b3b732c25aab4a5d38237b269b3c07"
server_key_version: "encryptionkey@0"
I20260928 12:25:28.871063 23610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "241bb66827904aaa990979165d6a7a4e"
format_stamp: "Formatted at 2026-09-28 12:25:28 on dist-test-slave-920p"
server_key: "91bc7f0b81f7669c743db391ec260da3"
server_key_iv: "36b3b732c25aab4a5d38237b269b3c07"
server_key_version: "encryptionkey@0"
I20260928 12:25:28.872190 23610 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:25:28.872772 23625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:28.872938 23610 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:28.873005 23610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "241bb66827904aaa990979165d6a7a4e"
format_stamp: "Formatted at 2026-09-28 12:25:28 on dist-test-slave-920p"
server_key: "91bc7f0b81f7669c743db391ec260da3"
server_key_iv: "36b3b732c25aab4a5d38237b269b3c07"
server_key_version: "encryptionkey@0"
I20260928 12:25:28.873056 23610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:28.886122 23610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:28.887720 23610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:28.887822 23610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:28.891393 23610 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43263
I20260928 12:25:28.891425 23677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43263 every 8 connection(s)
I20260928 12:25:28.891731 23610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:25:28.892503 23678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:28.893194 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 23610
I20260928 12:25:28.893311 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:25:28.893483 2272 external_mini_cluster.cc:1428] Setting key bb965521abdd4cb65e1799bbc60c2789
I20260928 12:25:28.895416 23678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 241bb66827904aaa990979165d6a7a4e: Bootstrap starting.
I20260928 12:25:28.896091 23678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 241bb66827904aaa990979165d6a7a4e: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:28.896379 23678 log.cc:824] T 00000000000000000000000000000000 P 241bb66827904aaa990979165d6a7a4e: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:28.896973 23678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 241bb66827904aaa990979165d6a7a4e: No bootstrap required, opened a new log
I20260928 12:25:28.899132 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:28.893917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.208:42745 (local address 127.0.0.1:43263)
0928 12:25:28.894113 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:28.894121 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:28.894152 (+ 31us) server_negotiation.cc:407] Connection header received
0928 12:25:28.894620 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:28.894628 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:28.894696 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:28.894838 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:28.894976 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:28.896262 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:28.897018 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:28.897696 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:28.897817 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:28.898409 (+ 592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:28.898413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:28.898413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:28.898609 (+ 196us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:28.898651 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:28.898664 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:28.898757 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:28.898872 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 12:25:28.898899 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":89,"threads_started":1}
I20260928 12:25:28.899339 23678 raft_consensus.cc:348] T 00000000000000000000000000000000 P 241bb66827904aaa990979165d6a7a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241bb66827904aaa990979165d6a7a4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43263 } }
I20260928 12:25:28.899469 23678 raft_consensus.cc:374] T 00000000000000000000000000000000 P 241bb66827904aaa990979165d6a7a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:28.899500 23678 raft_consensus.cc:729] T 00000000000000000000000000000000 P 241bb66827904aaa990979165d6a7a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 241bb66827904aaa990979165d6a7a4e, State: Initialized, Role: FOLLOWER
I20260928 12:25:28.899621 23678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 241bb66827904aaa990979165d6a7a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241bb66827904aaa990979165d6a7a4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43263 } }
I20260928 12:25:28.899694 23678 raft_consensus.cc:388] T 00000000000000000000000000000000 P 241bb66827904aaa990979165d6a7a4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:25:28.899721 23678 raft_consensus.cc:482] T 00000000000000000000000000000000 P 241bb66827904aaa990979165d6a7a4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:25:28.899767 23678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 241bb66827904aaa990979165d6a7a4e [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:28.900081 2272 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:25:28.900305 23678 raft_consensus.cc:504] T 00000000000000000000000000000000 P 241bb66827904aaa990979165d6a7a4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241bb66827904aaa990979165d6a7a4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43263 } }
I20260928 12:25:28.900395 23678 leader_election.cc:302] T 00000000000000000000000000000000 P 241bb66827904aaa990979165d6a7a4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 241bb66827904aaa990979165d6a7a4e; no voters:
I20260928 12:25:28.900558 23678 leader_election.cc:288] T 00000000000000000000000000000000 P 241bb66827904aaa990979165d6a7a4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:25:28.900586 23683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 241bb66827904aaa990979165d6a7a4e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:25:28.900755 23683 raft_consensus.cc:686] T 00000000000000000000000000000000 P 241bb66827904aaa990979165d6a7a4e [term 1 LEADER]: Becoming Leader. State: Replica: 241bb66827904aaa990979165d6a7a4e, State: Running, Role: LEADER
I20260928 12:25:28.900835 23683 consensus_queue.cc:237] T 00000000000000000000000000000000 P 241bb66827904aaa990979165d6a7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241bb66827904aaa990979165d6a7a4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43263 } }
I20260928 12:25:28.900863 23678 sys_catalog.cc:564] T 00000000000000000000000000000000 P 241bb66827904aaa990979165d6a7a4e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:25:28.901158 23683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 241bb66827904aaa990979165d6a7a4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "241bb66827904aaa990979165d6a7a4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241bb66827904aaa990979165d6a7a4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43263 } } }
I20260928 12:25:28.901225 23683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 241bb66827904aaa990979165d6a7a4e [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:28.901407 23683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 241bb66827904aaa990979165d6a7a4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 241bb66827904aaa990979165d6a7a4e. Latest consensus state: current_term: 1 leader_uuid: "241bb66827904aaa990979165d6a7a4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241bb66827904aaa990979165d6a7a4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43263 } } }
I20260928 12:25:28.901526 23683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 241bb66827904aaa990979165d6a7a4e [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:28.901650 23693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:25:28.902230 23693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:25:28.903824 23693 catalog_manager.cc:1347] Generated new cluster ID: 916498ff68ec490f9d7577db0f4c90ba
I20260928 12:25:28.903869 23693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:25:28.906352 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:28.901442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.208:35139 (local address 127.0.0.1:43263)
0928 12:25:28.901475 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:28.901477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:28.903027 (+ 1550us) server_negotiation.cc:407] Connection header received
0928 12:25:28.903054 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:28.903055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:28.903066 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:28.903134 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:28.903342 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:28.904046 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:28.905035 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:28.905381 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:28.905451 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:28.906104 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:28.906105 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:28.906106 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:28.906111 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:28.906128 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:28.906133 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:28.906196 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:28.906256 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 12:25:28.906266 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 12:25:28.911283 23693 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:25:28.911764 23693 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:25:28.917425 23693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 241bb66827904aaa990979165d6a7a4e: Generated new TSK 0
I20260928 12:25:28.917593 23693 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:25:28.924753 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 23610
W20260928 12:25:28.928356 23595 connection.cc:535] client connection to 127.0.0.1:43263 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43263): Connection reset by peer (error 104)
2026-09-28T12:25:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1196 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:25:28 dist-test-slave-920p krb5kdc[23709](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:25:28 dist-test-slave-920p krb5kdc[23709](info): set up 2 sockets
Sep 28 12:25:28 dist-test-slave-920p krb5kdc[23709](info): commencing operation
krb5kdc: starting...
W20260928 12:25:29.873178 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:25:29 dist-test-slave-920p krb5kdc[23709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:25:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:25:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 12:25:29.981976 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41915
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:25:30.060335 23722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:30.060532 23722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:30.060563 23722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:30.061710 23722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:25:30.061749 23722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:30.061767 23722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:30.061784 23722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:25:30.061800 23722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:25:30.063246 23722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41915
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598072638673-2272-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:30.063472 23722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:30.063663 23722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:30.068337 23727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:30.068477 23722 server_base.cc:1034] running on GCE node
W20260928 12:25:30.068349 23728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:30.068413 23730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:30.068781 23722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:30.069082 23722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:30.070218 23722 hybrid_clock.cc:648] HybridClock initialized: now 1790598330070205 us; error 28 us; skew 500 ppm
Sep 28 12:25:30 dist-test-slave-920p krb5kdc[23709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:30.076287 23722 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 12:25:30.076705 23722 webserver.cc:460] Webserver started at http://127.0.0.1:40355/ using document root <none> and password file <none>
I20260928 12:25:30.076918 23722 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:30.076965 23722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:30.077085 23722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:30.078123 23722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "dfb2b37697224333b98928af52bda8d9"
format_stamp: "Formatted at 2026-09-28 12:25:30 on dist-test-slave-920p"
server_key: "64a43bc183cc506eabf51089a31a2dd0"
server_key_iv: "9bb228ba35252f96affc0d4b90609c9f"
server_key_version: "encryptionkey@0"
I20260928 12:25:30.078456 23722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "dfb2b37697224333b98928af52bda8d9"
format_stamp: "Formatted at 2026-09-28 12:25:30 on dist-test-slave-920p"
server_key: "64a43bc183cc506eabf51089a31a2dd0"
server_key_iv: "9bb228ba35252f96affc0d4b90609c9f"
server_key_version: "encryptionkey@0"
I20260928 12:25:30.079622 23722 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:25:30.080255 23737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:30.080401 23722 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:25:30.080482 23722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "dfb2b37697224333b98928af52bda8d9"
format_stamp: "Formatted at 2026-09-28 12:25:30 on dist-test-slave-920p"
server_key: "64a43bc183cc506eabf51089a31a2dd0"
server_key_iv: "9bb228ba35252f96affc0d4b90609c9f"
server_key_version: "encryptionkey@0"
I20260928 12:25:30.080544 23722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:30.083632 23722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:30.085244 23722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:30.085388 23722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:30.088922 23722 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41915
I20260928 12:25:30.088951 23789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41915 every 8 connection(s)
I20260928 12:25:30.089249 23722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:25:30.090157 23790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:30.092155 23790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfb2b37697224333b98928af52bda8d9: Bootstrap starting.
I20260928 12:25:30.092653 23790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfb2b37697224333b98928af52bda8d9: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:30.092872 23790 log.cc:824] T 00000000000000000000000000000000 P dfb2b37697224333b98928af52bda8d9: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:30.093572 23790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfb2b37697224333b98928af52bda8d9: No bootstrap required, opened a new log
I20260928 12:25:30.095086 23790 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfb2b37697224333b98928af52bda8d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfb2b37697224333b98928af52bda8d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41915 } }
I20260928 12:25:30.095218 23790 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfb2b37697224333b98928af52bda8d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:30.095238 23790 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfb2b37697224333b98928af52bda8d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfb2b37697224333b98928af52bda8d9, State: Initialized, Role: FOLLOWER
I20260928 12:25:30.095346 23790 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfb2b37697224333b98928af52bda8d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfb2b37697224333b98928af52bda8d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41915 } }
I20260928 12:25:30.095417 23790 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfb2b37697224333b98928af52bda8d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:25:30.095449 23790 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfb2b37697224333b98928af52bda8d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:25:30.095479 23790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfb2b37697224333b98928af52bda8d9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:30.095942 23790 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfb2b37697224333b98928af52bda8d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfb2b37697224333b98928af52bda8d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41915 } }
I20260928 12:25:30.096020 23790 leader_election.cc:302] T 00000000000000000000000000000000 P dfb2b37697224333b98928af52bda8d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfb2b37697224333b98928af52bda8d9; no voters:
I20260928 12:25:30.096174 23790 leader_election.cc:288] T 00000000000000000000000000000000 P dfb2b37697224333b98928af52bda8d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:25:30.096182 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 23722
I20260928 12:25:30.096236 23793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfb2b37697224333b98928af52bda8d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:25:30.096278 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:25:30.096405 23790 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfb2b37697224333b98928af52bda8d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:25:30.096446 2272 external_mini_cluster.cc:1428] Setting key 4e8e11eba9e67a4481df3aa3893007fa
I20260928 12:25:30.096626 23793 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfb2b37697224333b98928af52bda8d9 [term 1 LEADER]: Becoming Leader. State: Replica: dfb2b37697224333b98928af52bda8d9, State: Running, Role: LEADER
I20260928 12:25:30.096740 23793 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfb2b37697224333b98928af52bda8d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfb2b37697224333b98928af52bda8d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41915 } }
I20260928 12:25:30.097170 23794 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfb2b37697224333b98928af52bda8d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfb2b37697224333b98928af52bda8d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfb2b37697224333b98928af52bda8d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41915 } } }
I20260928 12:25:30.097186 23795 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfb2b37697224333b98928af52bda8d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfb2b37697224333b98928af52bda8d9. Latest consensus state: current_term: 1 leader_uuid: "dfb2b37697224333b98928af52bda8d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfb2b37697224333b98928af52bda8d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41915 } } }
I20260928 12:25:30.097283 23794 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfb2b37697224333b98928af52bda8d9 [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:30.097298 23795 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfb2b37697224333b98928af52bda8d9 [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:30.097486 23804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:25:30.097977 23804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:25:30.099320 23799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:30.096903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.208:49677 (local address 127.0.0.1:41915)
0928 12:25:30.097519 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:30.097527 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:30.097561 (+ 34us) server_negotiation.cc:407] Connection header received
0928 12:25:30.097675 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:30.097681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:30.097713 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:30.098212 (+ 499us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:30.098805 (+ 593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:30.098807 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:30.098807 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:30.098811 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:30.098843 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:30.098855 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:30.098916 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:30.099004 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 12:25:30.099142 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":202,"threads_started":1}
I20260928 12:25:30.099644 23804 catalog_manager.cc:1347] Generated new cluster ID: e80ff0bd650d4150921cdef8982e4e84
I20260928 12:25:30.099687 23804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:25:30.100158 2272 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:25:30.102433 23799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:30.101373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.208:37773 (local address 127.0.0.1:41915)
0928 12:25:30.101401 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:30.101402 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:30.101637 (+ 235us) server_negotiation.cc:407] Connection header received
0928 12:25:30.101693 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:30.101694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:30.101698 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:30.101745 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:30.102259 (+ 514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:30.102260 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:30.102260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:30.102263 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:30.102275 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:30.102278 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:30.102317 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:30.102367 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 12:25:30.102377 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260928 12:25:30.131279 23804 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:25:30.131453 23804 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:25:30.138262 23804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfb2b37697224333b98928af52bda8d9: Generated new TSK 0
I20260928 12:25:30.138420 23804 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:25:30.143134 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 23722
2026-09-28T12:25:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1218 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:25:30 dist-test-slave-920p krb5kdc[23821](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:25:30 dist-test-slave-920p krb5kdc[23821](info): set up 2 sockets
Sep 28 12:25:30 dist-test-slave-920p krb5kdc[23821](info): commencing operation
krb5kdc: starting...
W20260928 12:25:31.097245 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:25:31 dist-test-slave-920p krb5kdc[23821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:25:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:25:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
I20260928 12:25:31.207417 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:42571
--webserver_interface=127.2.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:38121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:42571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:25:31.283244 23834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:31.283428 23834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:31.283457 23834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:31.284548 23834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:25:31.284595 23834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:31.284612 23834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:31.284627 23834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:25:31.284643 23834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:25:31.286096 23834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:38121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:42571
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:42571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:31.286320 23834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:31.286509 23834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:31.290822 23842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:31.290915 23834 server_base.cc:1034] running on GCE node
W20260928 12:25:31.290817 23839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:31.291029 23840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:31.291239 23834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:31.291483 23834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:31.292631 23834 hybrid_clock.cc:648] HybridClock initialized: now 1790598331292617 us; error 29 us; skew 500 ppm
Sep 28 12:25:31 dist-test-slave-920p krb5kdc[23821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598331, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:31.294957 23834 init.cc:377] Logged in from keytab as kudu/127.2.56.62@KRBTEST.COM (short username kudu)
I20260928 12:25:31.295315 23834 webserver.cc:460] Webserver started at http://127.2.56.62:35711/ using document root <none> and password file <none>
I20260928 12:25:31.295475 23834 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:31.295518 23834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:31.295603 23834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:31.296551 23834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "be217dd7e0f64f108b9fcf0a027762bc"
format_stamp: "Formatted at 2026-09-28 12:25:31 on dist-test-slave-920p"
server_key: "a1a167fc6cb29fabf5185071e166fc5f"
server_key_iv: "36d4d03ba28f2408ddc7acda0561ff8c"
server_key_version: "encryptionkey@0"
I20260928 12:25:31.296824 23834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "be217dd7e0f64f108b9fcf0a027762bc"
format_stamp: "Formatted at 2026-09-28 12:25:31 on dist-test-slave-920p"
server_key: "a1a167fc6cb29fabf5185071e166fc5f"
server_key_iv: "36d4d03ba28f2408ddc7acda0561ff8c"
server_key_version: "encryptionkey@0"
I20260928 12:25:31.298032 23834 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:25:31.298615 23849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:31.298753 23834 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:25:31.298823 23834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "be217dd7e0f64f108b9fcf0a027762bc"
format_stamp: "Formatted at 2026-09-28 12:25:31 on dist-test-slave-920p"
server_key: "a1a167fc6cb29fabf5185071e166fc5f"
server_key_iv: "36d4d03ba28f2408ddc7acda0561ff8c"
server_key_version: "encryptionkey@0"
I20260928 12:25:31.298887 23834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:31.327976 23834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:31.329972 23834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:31.330086 23834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:31.333663 23834 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:42571
I20260928 12:25:31.333702 23901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:42571 every 8 connection(s)
I20260928 12:25:31.334005 23834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:25:31.334828 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:31.336997 23902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be217dd7e0f64f108b9fcf0a027762bc: Bootstrap starting.
I20260928 12:25:31.337571 23902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be217dd7e0f64f108b9fcf0a027762bc: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:31.337816 23902 log.cc:824] T 00000000000000000000000000000000 P be217dd7e0f64f108b9fcf0a027762bc: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:31.338445 23902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be217dd7e0f64f108b9fcf0a027762bc: No bootstrap required, opened a new log
I20260928 12:25:31.339962 23902 raft_consensus.cc:348] T 00000000000000000000000000000000 P be217dd7e0f64f108b9fcf0a027762bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be217dd7e0f64f108b9fcf0a027762bc" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 42571 } }
I20260928 12:25:31.340097 23902 raft_consensus.cc:374] T 00000000000000000000000000000000 P be217dd7e0f64f108b9fcf0a027762bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:31.340129 23902 raft_consensus.cc:729] T 00000000000000000000000000000000 P be217dd7e0f64f108b9fcf0a027762bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be217dd7e0f64f108b9fcf0a027762bc, State: Initialized, Role: FOLLOWER
I20260928 12:25:31.340260 23902 consensus_queue.cc:260] T 00000000000000000000000000000000 P be217dd7e0f64f108b9fcf0a027762bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be217dd7e0f64f108b9fcf0a027762bc" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 42571 } }
I20260928 12:25:31.340337 23902 raft_consensus.cc:388] T 00000000000000000000000000000000 P be217dd7e0f64f108b9fcf0a027762bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:25:31.340371 23902 raft_consensus.cc:482] T 00000000000000000000000000000000 P be217dd7e0f64f108b9fcf0a027762bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:25:31.340416 23902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be217dd7e0f64f108b9fcf0a027762bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:31.341184 23902 raft_consensus.cc:504] T 00000000000000000000000000000000 P be217dd7e0f64f108b9fcf0a027762bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be217dd7e0f64f108b9fcf0a027762bc" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 42571 } }
I20260928 12:25:31.341306 23902 leader_election.cc:302] T 00000000000000000000000000000000 P be217dd7e0f64f108b9fcf0a027762bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be217dd7e0f64f108b9fcf0a027762bc; no voters:
I20260928 12:25:31.341449 23902 leader_election.cc:288] T 00000000000000000000000000000000 P be217dd7e0f64f108b9fcf0a027762bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:25:31.341487 23905 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be217dd7e0f64f108b9fcf0a027762bc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:25:31.341485 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 23834
I20260928 12:25:31.341585 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:25:31.341634 23905 raft_consensus.cc:686] T 00000000000000000000000000000000 P be217dd7e0f64f108b9fcf0a027762bc [term 1 LEADER]: Becoming Leader. State: Replica: be217dd7e0f64f108b9fcf0a027762bc, State: Running, Role: LEADER
I20260928 12:25:31.341699 23902 sys_catalog.cc:564] T 00000000000000000000000000000000 P be217dd7e0f64f108b9fcf0a027762bc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:25:31.341748 2272 external_mini_cluster.cc:1428] Setting key 8b8b4dd64698b581df327a5bcb4cd675
I20260928 12:25:31.341706 23905 consensus_queue.cc:237] T 00000000000000000000000000000000 P be217dd7e0f64f108b9fcf0a027762bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be217dd7e0f64f108b9fcf0a027762bc" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 42571 } }
I20260928 12:25:31.342442 23906 sys_catalog.cc:455] T 00000000000000000000000000000000 P be217dd7e0f64f108b9fcf0a027762bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be217dd7e0f64f108b9fcf0a027762bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be217dd7e0f64f108b9fcf0a027762bc" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 42571 } } }
I20260928 12:25:31.342563 23906 sys_catalog.cc:458] T 00000000000000000000000000000000 P be217dd7e0f64f108b9fcf0a027762bc [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:31.342425 23907 sys_catalog.cc:455] T 00000000000000000000000000000000 P be217dd7e0f64f108b9fcf0a027762bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader be217dd7e0f64f108b9fcf0a027762bc. Latest consensus state: current_term: 1 leader_uuid: "be217dd7e0f64f108b9fcf0a027762bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be217dd7e0f64f108b9fcf0a027762bc" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 42571 } } }
I20260928 12:25:31.342818 23907 sys_catalog.cc:458] T 00000000000000000000000000000000 P be217dd7e0f64f108b9fcf0a027762bc [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:31.343088 23917 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:25:31.343698 23917 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:25:31.344933 23917 catalog_manager.cc:1347] Generated new cluster ID: 385e1ba3a0914facae2e91232c02ba42
I20260928 12:25:31.344977 23917 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:25:31 dist-test-slave-920p krb5kdc[23821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:25:31.353116 23913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:31.342209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59439 (local address 127.2.56.62:42571)
0928 12:25:31.342405 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:31.342413 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:31.342443 (+ 30us) server_negotiation.cc:407] Connection header received
0928 12:25:31.342557 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:31.342565 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:31.342638 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:31.342772 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:31.343681 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:31.345012 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:31.346089 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:31.346696 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:31.346800 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:31.349285 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:31.349289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:31.349289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:31.349299 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:31.351575 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:31.351978 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:31.351980 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:31.351980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:31.352087 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:31.352299 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:31.352299 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:31.352299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:31.352566 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:31.352703 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:31.352813 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 12:25:31.352902 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":91,"threads_started":1}
I20260928 12:25:31.353981 2272 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:25:31.355290 23917 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:25:31.355751 23917 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:25:31.360646 23917 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be217dd7e0f64f108b9fcf0a027762bc: Generated new TSK 0
I20260928 12:25:31.360796 23917 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:25:31.364609 23913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:31.355350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35285 (local address 127.2.56.62:42571)
0928 12:25:31.355382 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:31.355384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:31.357517 (+ 2133us) server_negotiation.cc:407] Connection header received
0928 12:25:31.357563 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:31.357564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:31.357574 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:31.357621 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:31.358153 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:31.358924 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:31.359888 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:31.360759 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:31.360822 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:31.362148 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:31.362149 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:31.362150 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:31.362157 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:31.363588 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:31.363928 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:31.363929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:31.363929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:31.363986 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:31.364157 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:31.364157 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:31.364157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:31.364277 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:31.364376 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:31.364447 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 12:25:31.364515 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 12:25:31.366596 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 23834
W20260928 12:25:31.370174 23819 connection.cc:535] client connection to 127.2.56.62:42571 recv error: Network error: failed to read from TLS socket (remote: 127.2.56.62:42571): Connection reset by peer (error 104)
2026-09-28T12:25:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1223 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:25:31 dist-test-slave-920p krb5kdc[23933](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:25:31 dist-test-slave-920p krb5kdc[23933](info): set up 2 sockets
Sep 28 12:25:31 dist-test-slave-920p krb5kdc[23933](info): commencing operation
krb5kdc: starting...
W20260928 12:25:32.321159 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:25:32 dist-test-slave-920p krb5kdc[23933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:25:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:25:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
I20260928 12:25:32.423871 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:39711
--webserver_interface=127.2.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:44679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:39711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:25:32.500607 23946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:32.500797 23946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:32.500824 23946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:32.501953 23946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:25:32.501992 23946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:32.502020 23946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:32.502036 23946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:25:32.502056 23946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:25:32.503475 23946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:44679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:39711
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:39711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:32.503700 23946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:32.503885 23946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:32.508379 23954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:32.508386 23951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:32.508395 23952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:32.508580 23946 server_base.cc:1034] running on GCE node
I20260928 12:25:32.508827 23946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:32.509084 23946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:32.510231 23946 hybrid_clock.cc:648] HybridClock initialized: now 1790598332510206 us; error 38 us; skew 500 ppm
Sep 28 12:25:32 dist-test-slave-920p krb5kdc[23933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598332, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:32.512568 23946 init.cc:377] Logged in from keytab as kudu/127.2.56.62@KRBTEST.COM (short username kudu)
I20260928 12:25:32.512935 23946 webserver.cc:460] Webserver started at http://127.2.56.62:41279/ using document root <none> and password file <none>
I20260928 12:25:32.513113 23946 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:32.513150 23946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:32.513235 23946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:32.514194 23946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "133420c1e2624a3dbd4824bc2741a185"
format_stamp: "Formatted at 2026-09-28 12:25:32 on dist-test-slave-920p"
server_key: "8ac31e34710b09cbe442bd774ed35918"
server_key_iv: "957809e94f9b43f6535438935cbc2e40"
server_key_version: "encryptionkey@0"
I20260928 12:25:32.514464 23946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "133420c1e2624a3dbd4824bc2741a185"
format_stamp: "Formatted at 2026-09-28 12:25:32 on dist-test-slave-920p"
server_key: "8ac31e34710b09cbe442bd774ed35918"
server_key_iv: "957809e94f9b43f6535438935cbc2e40"
server_key_version: "encryptionkey@0"
I20260928 12:25:32.515600 23946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:25:32.516247 23961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:32.516415 23946 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:25:32.516491 23946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "133420c1e2624a3dbd4824bc2741a185"
format_stamp: "Formatted at 2026-09-28 12:25:32 on dist-test-slave-920p"
server_key: "8ac31e34710b09cbe442bd774ed35918"
server_key_iv: "957809e94f9b43f6535438935cbc2e40"
server_key_version: "encryptionkey@0"
I20260928 12:25:32.516546 23946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:32.538194 23946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:32.539866 23946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:32.539966 23946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:32.543452 23946 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:39711
I20260928 12:25:32.543484 24013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:39711 every 8 connection(s)
I20260928 12:25:32.543776 23946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:25:32.544590 24014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:32.546775 24014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 133420c1e2624a3dbd4824bc2741a185: Bootstrap starting.
I20260928 12:25:32.547317 24014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 133420c1e2624a3dbd4824bc2741a185: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:32.547565 24014 log.cc:824] T 00000000000000000000000000000000 P 133420c1e2624a3dbd4824bc2741a185: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:32.548130 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 23946
I20260928 12:25:32.548249 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:25:32.548298 24014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 133420c1e2624a3dbd4824bc2741a185: No bootstrap required, opened a new log
I20260928 12:25:32.548434 2272 external_mini_cluster.cc:1428] Setting key a0e9341e5b2123e1ce68975d64f97332
I20260928 12:25:32.550768 24014 raft_consensus.cc:348] T 00000000000000000000000000000000 P 133420c1e2624a3dbd4824bc2741a185 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133420c1e2624a3dbd4824bc2741a185" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39711 } }
I20260928 12:25:32.550920 24014 raft_consensus.cc:374] T 00000000000000000000000000000000 P 133420c1e2624a3dbd4824bc2741a185 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:32.550956 24014 raft_consensus.cc:729] T 00000000000000000000000000000000 P 133420c1e2624a3dbd4824bc2741a185 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 133420c1e2624a3dbd4824bc2741a185, State: Initialized, Role: FOLLOWER
I20260928 12:25:32.551084 24014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 133420c1e2624a3dbd4824bc2741a185 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133420c1e2624a3dbd4824bc2741a185" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39711 } }
I20260928 12:25:32.551159 24014 raft_consensus.cc:388] T 00000000000000000000000000000000 P 133420c1e2624a3dbd4824bc2741a185 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:25:32.551196 24014 raft_consensus.cc:482] T 00000000000000000000000000000000 P 133420c1e2624a3dbd4824bc2741a185 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:25:32.551239 24014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 133420c1e2624a3dbd4824bc2741a185 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:32.551872 24014 raft_consensus.cc:504] T 00000000000000000000000000000000 P 133420c1e2624a3dbd4824bc2741a185 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133420c1e2624a3dbd4824bc2741a185" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39711 } }
I20260928 12:25:32.551959 24014 leader_election.cc:302] T 00000000000000000000000000000000 P 133420c1e2624a3dbd4824bc2741a185 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 133420c1e2624a3dbd4824bc2741a185; no voters:
I20260928 12:25:32.552109 24014 leader_election.cc:288] T 00000000000000000000000000000000 P 133420c1e2624a3dbd4824bc2741a185 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:25:32.552140 24019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 133420c1e2624a3dbd4824bc2741a185 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:25:32.552300 24019 raft_consensus.cc:686] T 00000000000000000000000000000000 P 133420c1e2624a3dbd4824bc2741a185 [term 1 LEADER]: Becoming Leader. State: Replica: 133420c1e2624a3dbd4824bc2741a185, State: Running, Role: LEADER
I20260928 12:25:32.552351 24014 sys_catalog.cc:564] T 00000000000000000000000000000000 P 133420c1e2624a3dbd4824bc2741a185 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:25:32.553586 24019 consensus_queue.cc:237] T 00000000000000000000000000000000 P 133420c1e2624a3dbd4824bc2741a185 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133420c1e2624a3dbd4824bc2741a185" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39711 } }
I20260928 12:25:32.553814 24018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:32.548922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38549 (local address 127.2.56.62:39711)
0928 12:25:32.549091 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:32.549099 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:32.549125 (+ 26us) server_negotiation.cc:407] Connection header received
0928 12:25:32.549597 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:32.549603 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:32.549649 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:32.549752 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:32.549948 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:32.550771 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:32.551831 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:32.552369 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:32.552496 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:32.553188 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:32.553191 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:32.553192 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:32.553403 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:32.553437 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:32.553447 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:32.553517 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:32.553620 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 12:25:32.553646 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
I20260928 12:25:32.554574 24020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 133420c1e2624a3dbd4824bc2741a185 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 133420c1e2624a3dbd4824bc2741a185. Latest consensus state: current_term: 1 leader_uuid: "133420c1e2624a3dbd4824bc2741a185" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133420c1e2624a3dbd4824bc2741a185" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39711 } } }
I20260928 12:25:32.554665 24020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 133420c1e2624a3dbd4824bc2741a185 [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:32.554589 24021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 133420c1e2624a3dbd4824bc2741a185 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "133420c1e2624a3dbd4824bc2741a185" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133420c1e2624a3dbd4824bc2741a185" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 39711 } } }
I20260928 12:25:32.554767 24021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 133420c1e2624a3dbd4824bc2741a185 [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:32.554828 2272 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 12:25:32.555128 24033 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 133420c1e2624a3dbd4824bc2741a185: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:25:32.555159 24033 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:25:32.555214 24035 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:25:32.555361 24035 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:25:32.556820 24035 catalog_manager.cc:1347] Generated new cluster ID: 4d127bccc6204516a7fb0305d82bcc93
I20260928 12:25:32.556872 24035 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:25:32.558907 24018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:32.556279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43477 (local address 127.2.56.62:39711)
0928 12:25:32.556311 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:32.556313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:32.556424 (+ 111us) server_negotiation.cc:407] Connection header received
0928 12:25:32.556478 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:32.556479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:32.556487 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:32.556532 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:32.556640 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:32.557183 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:32.557889 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:32.558107 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:32.558161 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:32.558691 (+ 530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:32.558692 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:32.558693 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:32.558697 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:32.558711 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:32.558715 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:32.558763 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:32.558828 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 12:25:32.558837 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 12:25:32.576642 24035 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:25:32.577167 24035 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:25:32.584388 24035 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 133420c1e2624a3dbd4824bc2741a185: Generated new TSK 0
I20260928 12:25:32.584560 24035 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:25:32.593743 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 23946
W20260928 12:25:32.597340 23931 connection.cc:535] client connection to 127.2.56.62:39711 recv error: Network error: failed to read from TLS socket (remote: 127.2.56.62:39711): Connection reset by peer (error 104)
2026-09-28T12:25:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1227 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:25:32 dist-test-slave-920p krb5kdc[24045](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:25:32 dist-test-slave-920p krb5kdc[24045](info): set up 2 sockets
Sep 28 12:25:32 dist-test-slave-920p krb5kdc[24045](info): commencing operation
krb5kdc: starting...
W20260928 12:25:33.549352 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:25:33 dist-test-slave-920p krb5kdc[24045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:25:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:25:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
I20260928 12:25:33.665410 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:37153
--webserver_interface=127.2.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:37153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:25:33.742177 24058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:33.742364 24058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:33.742385 24058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:33.743494 24058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:25:33.743533 24058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:33.743546 24058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:33.743557 24058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:25:33.743579 24058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:25:33.745011 24058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:37153
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:37153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:33.745235 24058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:33.745441 24058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:33.749851 24066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:33.749871 24063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:33.749867 24064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:33.750038 24058 server_base.cc:1034] running on GCE node
I20260928 12:25:33.750289 24058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:33.750559 24058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:33.751695 24058 hybrid_clock.cc:648] HybridClock initialized: now 1790598333751683 us; error 27 us; skew 500 ppm
Sep 28 12:25:33 dist-test-slave-920p krb5kdc[24045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598333, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:33.753929 24058 init.cc:377] Logged in from keytab as kudu/127.2.56.62@KRBTEST.COM (short username kudu)
I20260928 12:25:33.754313 24058 webserver.cc:460] Webserver started at http://127.2.56.62:45813/ using document root <none> and password file <none>
I20260928 12:25:33.754509 24058 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:33.754554 24058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:33.754673 24058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:33.755645 24058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "484c61ef1b6b4dfd937698d0fa0cc768"
format_stamp: "Formatted at 2026-09-28 12:25:33 on dist-test-slave-920p"
server_key: "bb3839d1cc720e86930a7a1df5d2bdc3"
server_key_iv: "5cac14646b04cf8e2d76fe9f6d9015c6"
server_key_version: "encryptionkey@0"
I20260928 12:25:33.755945 24058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "484c61ef1b6b4dfd937698d0fa0cc768"
format_stamp: "Formatted at 2026-09-28 12:25:33 on dist-test-slave-920p"
server_key: "bb3839d1cc720e86930a7a1df5d2bdc3"
server_key_iv: "5cac14646b04cf8e2d76fe9f6d9015c6"
server_key_version: "encryptionkey@0"
I20260928 12:25:33.757084 24058 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:25:33.757813 24073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:33.757970 24058 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:33.758046 24058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "484c61ef1b6b4dfd937698d0fa0cc768"
format_stamp: "Formatted at 2026-09-28 12:25:33 on dist-test-slave-920p"
server_key: "bb3839d1cc720e86930a7a1df5d2bdc3"
server_key_iv: "5cac14646b04cf8e2d76fe9f6d9015c6"
server_key_version: "encryptionkey@0"
I20260928 12:25:33.758109 24058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:33.761178 24058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:33.763038 24058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:33.763155 24058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:33.766590 24058 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:37153
I20260928 12:25:33.766636 24125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:37153 every 8 connection(s)
I20260928 12:25:33.766901 24058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:25:33.767791 24126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:33.769327 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 24058
I20260928 12:25:33.769423 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:25:33.769580 2272 external_mini_cluster.cc:1428] Setting key 911213fbe65824acb9205037dff897e9
I20260928 12:25:33.770025 24126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 484c61ef1b6b4dfd937698d0fa0cc768: Bootstrap starting.
I20260928 12:25:33.770551 24126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 484c61ef1b6b4dfd937698d0fa0cc768: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:33.770794 24126 log.cc:824] T 00000000000000000000000000000000 P 484c61ef1b6b4dfd937698d0fa0cc768: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:33.771436 24126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 484c61ef1b6b4dfd937698d0fa0cc768: No bootstrap required, opened a new log
I20260928 12:25:33.771833 24129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:33.770000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37565 (local address 127.2.56.62:37153)
0928 12:25:33.770210 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:33.770216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:33.770240 (+ 24us) server_negotiation.cc:407] Connection header received
0928 12:25:33.770648 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:33.770655 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:33.770682 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:33.770799 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:33.771434 (+ 635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:33.771438 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:33.771438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:33.771441 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:33.771485 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:33.771501 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:33.771595 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:33.771654 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 12:25:33.771691 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":73,"threads_started":1}
I20260928 12:25:33.772634 2272 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:25:33.773559 24126 raft_consensus.cc:348] T 00000000000000000000000000000000 P 484c61ef1b6b4dfd937698d0fa0cc768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484c61ef1b6b4dfd937698d0fa0cc768" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 37153 } }
I20260928 12:25:33.773702 24126 raft_consensus.cc:374] T 00000000000000000000000000000000 P 484c61ef1b6b4dfd937698d0fa0cc768 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:33.773736 24126 raft_consensus.cc:729] T 00000000000000000000000000000000 P 484c61ef1b6b4dfd937698d0fa0cc768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 484c61ef1b6b4dfd937698d0fa0cc768, State: Initialized, Role: FOLLOWER
I20260928 12:25:33.773840 24126 consensus_queue.cc:260] T 00000000000000000000000000000000 P 484c61ef1b6b4dfd937698d0fa0cc768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484c61ef1b6b4dfd937698d0fa0cc768" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 37153 } }
I20260928 12:25:33.773924 24126 raft_consensus.cc:388] T 00000000000000000000000000000000 P 484c61ef1b6b4dfd937698d0fa0cc768 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:25:33.773957 24126 raft_consensus.cc:482] T 00000000000000000000000000000000 P 484c61ef1b6b4dfd937698d0fa0cc768 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:25:33.774003 24126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 484c61ef1b6b4dfd937698d0fa0cc768 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:33.774540 24126 raft_consensus.cc:504] T 00000000000000000000000000000000 P 484c61ef1b6b4dfd937698d0fa0cc768 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484c61ef1b6b4dfd937698d0fa0cc768" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 37153 } }
I20260928 12:25:33.774637 24126 leader_election.cc:302] T 00000000000000000000000000000000 P 484c61ef1b6b4dfd937698d0fa0cc768 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 484c61ef1b6b4dfd937698d0fa0cc768; no voters:
I20260928 12:25:33.774793 24126 leader_election.cc:288] T 00000000000000000000000000000000 P 484c61ef1b6b4dfd937698d0fa0cc768 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:25:33.774827 24136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 484c61ef1b6b4dfd937698d0fa0cc768 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:25:33.774909 24129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:33.773931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38005 (local address 127.2.56.62:37153)
0928 12:25:33.773964 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:33.773965 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:33.774068 (+ 103us) server_negotiation.cc:407] Connection header received
0928 12:25:33.774123 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:33.774124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:33.774128 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:33.774188 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:33.774617 (+ 429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:33.774619 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:33.774619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:33.774623 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:33.774635 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:33.774639 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:33.774704 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:33.774756 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 12:25:33.774767 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 12:25:33.775015 24136 raft_consensus.cc:686] T 00000000000000000000000000000000 P 484c61ef1b6b4dfd937698d0fa0cc768 [term 1 LEADER]: Becoming Leader. State: Replica: 484c61ef1b6b4dfd937698d0fa0cc768, State: Running, Role: LEADER
I20260928 12:25:33.775075 24126 sys_catalog.cc:564] T 00000000000000000000000000000000 P 484c61ef1b6b4dfd937698d0fa0cc768 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:25:33.775126 24136 consensus_queue.cc:237] T 00000000000000000000000000000000 P 484c61ef1b6b4dfd937698d0fa0cc768 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484c61ef1b6b4dfd937698d0fa0cc768" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 37153 } }
I20260928 12:25:33.775473 24138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 484c61ef1b6b4dfd937698d0fa0cc768 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "484c61ef1b6b4dfd937698d0fa0cc768" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484c61ef1b6b4dfd937698d0fa0cc768" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 37153 } } }
I20260928 12:25:33.775476 24137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 484c61ef1b6b4dfd937698d0fa0cc768 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 484c61ef1b6b4dfd937698d0fa0cc768. Latest consensus state: current_term: 1 leader_uuid: "484c61ef1b6b4dfd937698d0fa0cc768" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484c61ef1b6b4dfd937698d0fa0cc768" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 37153 } } }
I20260928 12:25:33.775610 24138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 484c61ef1b6b4dfd937698d0fa0cc768 [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:33.775627 24137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 484c61ef1b6b4dfd937698d0fa0cc768 [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:33.775995 24143 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:25:33.776455 24143 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:25:33.777812 24143 catalog_manager.cc:1347] Generated new cluster ID: 28339f9fab444c2284d9c0c93e869857
I20260928 12:25:33.777863 24143 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:25:33.786338 24143 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:25:33.786486 24143 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:25:33.790879 24143 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 484c61ef1b6b4dfd937698d0fa0cc768: Generated new TSK 0
I20260928 12:25:33.791024 24143 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:25:33.794101 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 24058
2026-09-28T12:25:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1200 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:25:33 dist-test-slave-920p krb5kdc[24157](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:25:33 dist-test-slave-920p krb5kdc[24157](info): set up 2 sockets
Sep 28 12:25:33 dist-test-slave-920p krb5kdc[24157](info): commencing operation
krb5kdc: starting...
W20260928 12:25:34.745330 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:25:34 dist-test-slave-920p krb5kdc[24157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:25:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:25:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
I20260928 12:25:34.866269 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:33523
--webserver_interface=127.2.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:34699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:33523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:25:34.956099 24170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:34.956393 24170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:34.956420 24170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:34.957752 24170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:25:34.957810 24170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:34.957826 24170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:34.957840 24170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:25:34.957854 24170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:25:34.959491 24170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:34699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:33523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:33523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:34.959745 24170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:34.959971 24170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:34.965467 24178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:34.965482 24176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:34.965476 24175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:34.965777 24170 server_base.cc:1034] running on GCE node
I20260928 12:25:34.966059 24170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:34.966375 24170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:34.967582 24170 hybrid_clock.cc:648] HybridClock initialized: now 1790598334967555 us; error 50 us; skew 500 ppm
Sep 28 12:25:34 dist-test-slave-920p krb5kdc[24157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598334, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:34.971175 24170 init.cc:377] Logged in from keytab as kudu/127.2.56.62@KRBTEST.COM (short username kudu)
I20260928 12:25:34.971719 24170 webserver.cc:460] Webserver started at http://127.2.56.62:40885/ using document root <none> and password file <none>
I20260928 12:25:34.971957 24170 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:34.972012 24170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:34.972144 24170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:34.973317 24170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "bde44fd128774354a420772d6b8ba68a"
format_stamp: "Formatted at 2026-09-28 12:25:34 on dist-test-slave-920p"
server_key: "abb473bcc0a029f4963ae81d63122eb1"
server_key_iv: "5654e75aa16e54de215c1d6538f5dba2"
server_key_version: "encryptionkey@0"
I20260928 12:25:34.973685 24170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "bde44fd128774354a420772d6b8ba68a"
format_stamp: "Formatted at 2026-09-28 12:25:34 on dist-test-slave-920p"
server_key: "abb473bcc0a029f4963ae81d63122eb1"
server_key_iv: "5654e75aa16e54de215c1d6538f5dba2"
server_key_version: "encryptionkey@0"
I20260928 12:25:34.975135 24170 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:25:34.975945 24185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:34.976152 24170 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:25:34.976254 24170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "bde44fd128774354a420772d6b8ba68a"
format_stamp: "Formatted at 2026-09-28 12:25:34 on dist-test-slave-920p"
server_key: "abb473bcc0a029f4963ae81d63122eb1"
server_key_iv: "5654e75aa16e54de215c1d6538f5dba2"
server_key_version: "encryptionkey@0"
I20260928 12:25:34.976317 24170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:34.997813 24170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:34.999812 24170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:34.999922 24170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:35.004073 24170 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:33523
I20260928 12:25:35.004158 24237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:33523 every 8 connection(s)
I20260928 12:25:35.004475 24170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:25:35.005537 24238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:35.008270 24238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bde44fd128774354a420772d6b8ba68a: Bootstrap starting.
I20260928 12:25:35.008920 24238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bde44fd128774354a420772d6b8ba68a: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:35.009205 24238 log.cc:824] T 00000000000000000000000000000000 P bde44fd128774354a420772d6b8ba68a: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:35.010139 24238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bde44fd128774354a420772d6b8ba68a: No bootstrap required, opened a new log
I20260928 12:25:35.011996 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 24170
I20260928 12:25:35.012115 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:25:35.012097 24238 raft_consensus.cc:348] T 00000000000000000000000000000000 P bde44fd128774354a420772d6b8ba68a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde44fd128774354a420772d6b8ba68a" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 33523 } }
I20260928 12:25:35.012317 24238 raft_consensus.cc:374] T 00000000000000000000000000000000 P bde44fd128774354a420772d6b8ba68a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:35.012336 2272 external_mini_cluster.cc:1428] Setting key 819e5996ea8a03debc10c2374938049b
I20260928 12:25:35.012363 24238 raft_consensus.cc:729] T 00000000000000000000000000000000 P bde44fd128774354a420772d6b8ba68a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bde44fd128774354a420772d6b8ba68a, State: Initialized, Role: FOLLOWER
I20260928 12:25:35.012522 24238 consensus_queue.cc:260] T 00000000000000000000000000000000 P bde44fd128774354a420772d6b8ba68a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde44fd128774354a420772d6b8ba68a" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 33523 } }
I20260928 12:25:35.012624 24238 raft_consensus.cc:388] T 00000000000000000000000000000000 P bde44fd128774354a420772d6b8ba68a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:25:35.012655 24238 raft_consensus.cc:482] T 00000000000000000000000000000000 P bde44fd128774354a420772d6b8ba68a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:25:35.012709 24238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bde44fd128774354a420772d6b8ba68a [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:35.013972 24238 raft_consensus.cc:504] T 00000000000000000000000000000000 P bde44fd128774354a420772d6b8ba68a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde44fd128774354a420772d6b8ba68a" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 33523 } }
I20260928 12:25:35.014079 24238 leader_election.cc:302] T 00000000000000000000000000000000 P bde44fd128774354a420772d6b8ba68a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bde44fd128774354a420772d6b8ba68a; no voters:
I20260928 12:25:35.014264 24238 leader_election.cc:288] T 00000000000000000000000000000000 P bde44fd128774354a420772d6b8ba68a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:25:35.014339 24243 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bde44fd128774354a420772d6b8ba68a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:25:35.014529 24243 raft_consensus.cc:686] T 00000000000000000000000000000000 P bde44fd128774354a420772d6b8ba68a [term 1 LEADER]: Becoming Leader. State: Replica: bde44fd128774354a420772d6b8ba68a, State: Running, Role: LEADER
I20260928 12:25:35.014642 24238 sys_catalog.cc:564] T 00000000000000000000000000000000 P bde44fd128774354a420772d6b8ba68a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:25:35.014709 24243 consensus_queue.cc:237] T 00000000000000000000000000000000 P bde44fd128774354a420772d6b8ba68a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde44fd128774354a420772d6b8ba68a" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 33523 } }
I20260928 12:25:35.015224 24243 sys_catalog.cc:455] T 00000000000000000000000000000000 P bde44fd128774354a420772d6b8ba68a [sys.catalog]: SysCatalogTable state changed. Reason: New leader bde44fd128774354a420772d6b8ba68a. Latest consensus state: current_term: 1 leader_uuid: "bde44fd128774354a420772d6b8ba68a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde44fd128774354a420772d6b8ba68a" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 33523 } } }
I20260928 12:25:35.015236 24245 sys_catalog.cc:455] T 00000000000000000000000000000000 P bde44fd128774354a420772d6b8ba68a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bde44fd128774354a420772d6b8ba68a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde44fd128774354a420772d6b8ba68a" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 33523 } } }
I20260928 12:25:35.015348 24243 sys_catalog.cc:458] T 00000000000000000000000000000000 P bde44fd128774354a420772d6b8ba68a [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:35.015426 24245 sys_catalog.cc:458] T 00000000000000000000000000000000 P bde44fd128774354a420772d6b8ba68a [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:35.015635 24252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:25:35.016314 24252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:25:35.018534 24252 catalog_manager.cc:1347] Generated new cluster ID: dab1636add424effa6dbc3c0435f2e29
I20260928 12:25:35.018688 24252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:25:35 dist-test-slave-920p krb5kdc[24157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:25:35.024240 24242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:35.012948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36125 (local address 127.2.56.62:33523)
0928 12:25:35.013178 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:35.013189 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:35.013231 (+ 42us) server_negotiation.cc:407] Connection header received
0928 12:25:35.013418 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:35.013427 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:35.013501 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:35.013685 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:35.014364 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:35.015903 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:35.017088 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:35.017772 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:35.017930 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:35.020682 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:35.020686 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:35.020686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:35.020700 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:35.022398 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:35.022744 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:35.022747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:35.022748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:35.022849 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:35.023025 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:35.023026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:35.023027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:35.023299 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:35.023876 (+ 577us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:35.023939 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 12:25:35.024064 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":107,"threads_started":1}
I20260928 12:25:35.025327 2272 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:25:35.031520 24252 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:25:35.032236 24252 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:25:35.035660 24252 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bde44fd128774354a420772d6b8ba68a: Generated new TSK 0
I20260928 12:25:35.036152 24252 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:25:35.038358 24242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:35.031421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45057 (local address 127.2.56.62:33523)
0928 12:25:35.031485 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:35.031487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:35.031802 (+ 315us) server_negotiation.cc:407] Connection header received
0928 12:25:35.031864 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:35.031865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:35.031877 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:35.031927 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:35.032462 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:35.033046 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:35.033804 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:35.034022 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:35.034077 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:35.035458 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:35.035461 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:35.035462 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:35.035471 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:35.037158 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:35.037576 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:35.037578 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:35.037578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:35.037642 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:35.037827 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:35.037828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:35.037828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:35.037964 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:35.038068 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:35.038147 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 12:25:35.038263 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 12:25:35.040356 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 24170
2026-09-28T12:25:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1246 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:25:35 dist-test-slave-920p krb5kdc[24269](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:25:35 dist-test-slave-920p krb5kdc[24269](info): set up 2 sockets
Sep 28 12:25:35 dist-test-slave-920p krb5kdc[24269](info): commencing operation
krb5kdc: starting...
W20260928 12:25:35.997113 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:25:36 dist-test-slave-920p krb5kdc[24269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:25:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:25:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
I20260928 12:25:36.107827 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:37259
--webserver_interface=127.2.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:44157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:37259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:25:36.188165 24282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:36.188357 24282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:36.188385 24282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:36.189500 24282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:25:36.189538 24282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:36.189555 24282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:36.189571 24282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:25:36.189586 24282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:25:36.191007 24282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:44157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:37259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:37259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:36.191239 24282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:36.191452 24282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:36.195891 24288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:36.195863 24287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:36.195871 24290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:36.196024 24282 server_base.cc:1034] running on GCE node
I20260928 12:25:36.196346 24282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:36.196617 24282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:36.197749 24282 hybrid_clock.cc:648] HybridClock initialized: now 1790598336197736 us; error 25 us; skew 500 ppm
Sep 28 12:25:36 dist-test-slave-920p krb5kdc[24269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598336, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:36.199908 24282 init.cc:377] Logged in from keytab as kudu/127.2.56.62@KRBTEST.COM (short username kudu)
I20260928 12:25:36.200273 24282 webserver.cc:460] Webserver started at http://127.2.56.62:38859/ using document root <none> and password file <none>
I20260928 12:25:36.200474 24282 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:36.200518 24282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:36.200642 24282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:36.201689 24282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "bd576e87409b4ebcb64b264fd34b2b29"
format_stamp: "Formatted at 2026-09-28 12:25:36 on dist-test-slave-920p"
server_key: "7a0e32d735ad2ef35d19ff502ea35910"
server_key_iv: "b7caf21e31e3e86625dbc210d2077e22"
server_key_version: "encryptionkey@0"
I20260928 12:25:36.201992 24282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "bd576e87409b4ebcb64b264fd34b2b29"
format_stamp: "Formatted at 2026-09-28 12:25:36 on dist-test-slave-920p"
server_key: "7a0e32d735ad2ef35d19ff502ea35910"
server_key_iv: "b7caf21e31e3e86625dbc210d2077e22"
server_key_version: "encryptionkey@0"
I20260928 12:25:36.203130 24282 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:25:36.203737 24297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:36.203871 24282 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:25:36.203948 24282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "bd576e87409b4ebcb64b264fd34b2b29"
format_stamp: "Formatted at 2026-09-28 12:25:36 on dist-test-slave-920p"
server_key: "7a0e32d735ad2ef35d19ff502ea35910"
server_key_iv: "b7caf21e31e3e86625dbc210d2077e22"
server_key_version: "encryptionkey@0"
I20260928 12:25:36.204006 24282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:36.238492 24282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:36.240204 24282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:36.240332 24282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:36.243913 24282 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:37259
I20260928 12:25:36.243956 24349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:37259 every 8 connection(s)
I20260928 12:25:36.244259 24282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:25:36.245069 24350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:36.247241 24350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd576e87409b4ebcb64b264fd34b2b29: Bootstrap starting.
I20260928 12:25:36.247730 24350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd576e87409b4ebcb64b264fd34b2b29: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:36.247946 24350 log.cc:824] T 00000000000000000000000000000000 P bd576e87409b4ebcb64b264fd34b2b29: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:36.248517 24350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd576e87409b4ebcb64b264fd34b2b29: No bootstrap required, opened a new log
I20260928 12:25:36.250051 24350 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd576e87409b4ebcb64b264fd34b2b29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd576e87409b4ebcb64b264fd34b2b29" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 37259 } }
I20260928 12:25:36.250177 24350 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd576e87409b4ebcb64b264fd34b2b29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:36.250198 24350 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd576e87409b4ebcb64b264fd34b2b29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd576e87409b4ebcb64b264fd34b2b29, State: Initialized, Role: FOLLOWER
I20260928 12:25:36.250303 24350 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd576e87409b4ebcb64b264fd34b2b29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd576e87409b4ebcb64b264fd34b2b29" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 37259 } }
I20260928 12:25:36.250373 24350 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd576e87409b4ebcb64b264fd34b2b29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:25:36.250401 24350 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd576e87409b4ebcb64b264fd34b2b29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:25:36.250432 24350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd576e87409b4ebcb64b264fd34b2b29 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:36.251144 24350 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd576e87409b4ebcb64b264fd34b2b29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd576e87409b4ebcb64b264fd34b2b29" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 37259 } }
I20260928 12:25:36.251226 24350 leader_election.cc:302] T 00000000000000000000000000000000 P bd576e87409b4ebcb64b264fd34b2b29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd576e87409b4ebcb64b264fd34b2b29; no voters:
I20260928 12:25:36.251334 24350 leader_election.cc:288] T 00000000000000000000000000000000 P bd576e87409b4ebcb64b264fd34b2b29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:25:36.251380 24353 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd576e87409b4ebcb64b264fd34b2b29 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:25:36.251529 24350 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd576e87409b4ebcb64b264fd34b2b29 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:25:36.251564 24353 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd576e87409b4ebcb64b264fd34b2b29 [term 1 LEADER]: Becoming Leader. State: Replica: bd576e87409b4ebcb64b264fd34b2b29, State: Running, Role: LEADER
I20260928 12:25:36.251677 24353 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd576e87409b4ebcb64b264fd34b2b29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd576e87409b4ebcb64b264fd34b2b29" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 37259 } }
I20260928 12:25:36.252229 24355 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd576e87409b4ebcb64b264fd34b2b29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd576e87409b4ebcb64b264fd34b2b29. Latest consensus state: current_term: 1 leader_uuid: "bd576e87409b4ebcb64b264fd34b2b29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd576e87409b4ebcb64b264fd34b2b29" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 37259 } } }
I20260928 12:25:36.252313 24355 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd576e87409b4ebcb64b264fd34b2b29 [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:36.252321 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 24282
I20260928 12:25:36.252422 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:25:36.252609 2272 external_mini_cluster.cc:1428] Setting key 502418fd1f8704d97733d57a0489733a
I20260928 12:25:36.252599 24354 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd576e87409b4ebcb64b264fd34b2b29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd576e87409b4ebcb64b264fd34b2b29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd576e87409b4ebcb64b264fd34b2b29" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 37259 } } }
I20260928 12:25:36.252645 24362 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:25:36.252669 24354 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd576e87409b4ebcb64b264fd34b2b29 [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:36.253288 24362 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:25:36.254733 24362 catalog_manager.cc:1347] Generated new cluster ID: b96e71d5e8214e758f2875844f5cf7de
I20260928 12:25:36.254771 24362 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:25:36.258165 24372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:36.253035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32949 (local address 127.2.56.62:37259)
0928 12:25:36.253201 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:36.253210 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:36.253241 (+ 31us) server_negotiation.cc:407] Connection header received
0928 12:25:36.253382 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:36.253390 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:36.253446 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:36.253589 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:36.253791 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:36.255105 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:36.256192 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:36.256765 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:36.256855 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:36.257538 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:36.257542 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:36.257542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:36.257730 (+ 188us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:36.257766 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:36.257779 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:36.257838 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:36.257910 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 12:25:36.257969 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
I20260928 12:25:36.259059 2272 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:25:36.263636 24372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:36.260508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39595 (local address 127.2.56.62:37259)
0928 12:25:36.260543 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:36.260545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:36.260552 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:25:36.260633 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:36.260634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:36.260644 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:36.260706 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:36.260804 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:36.261576 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:36.262502 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:36.262795 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:36.262863 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:36.263421 (+ 558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:36.263422 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:36.263423 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:36.263428 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:36.263442 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:36.263447 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:36.263499 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:36.263541 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 12:25:36.263564 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:25:36.272753 24362 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:25:36.273221 24362 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:25:36.279155 24362 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd576e87409b4ebcb64b264fd34b2b29: Generated new TSK 0
I20260928 12:25:36.279307 24362 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:25:36.287312 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 24282
2026-09-28T12:25:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1246 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:25:36 dist-test-slave-920p krb5kdc[24381](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:25:36 dist-test-slave-920p krb5kdc[24381](info): set up 2 sockets
Sep 28 12:25:36 dist-test-slave-920p krb5kdc[24381](info): commencing operation
krb5kdc: starting...
W20260928 12:25:37.241164 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:25:37 dist-test-slave-920p krb5kdc[24381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:25:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:25:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
I20260928 12:25:37.347978 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:45739
--webserver_interface=127.2.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:39791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:45739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:25:37.424086 24394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:37.424276 24394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:37.424305 24394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:37.425451 24394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:25:37.425491 24394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:37.425509 24394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:37.425525 24394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:25:37.425541 24394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:25:37.426970 24394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:39791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:45739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:45739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:37.427192 24394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:37.427386 24394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:37.431831 24399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:37.431886 24394 server_base.cc:1034] running on GCE node
W20260928 12:25:37.431856 24400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:37.431843 24402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:37.432281 24394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:37.432535 24394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:37.433665 24394 hybrid_clock.cc:648] HybridClock initialized: now 1790598337433652 us; error 28 us; skew 500 ppm
Sep 28 12:25:37 dist-test-slave-920p krb5kdc[24381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598337, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:37.435856 24394 init.cc:377] Logged in from keytab as kudu/127.2.56.62@KRBTEST.COM (short username kudu)
I20260928 12:25:37.436244 24394 webserver.cc:460] Webserver started at http://127.2.56.62:44043/ using document root <none> and password file <none>
I20260928 12:25:37.436422 24394 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:37.436462 24394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:37.436571 24394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:37.437572 24394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "6b4b7e7660354b8995fe022491a42a9e"
format_stamp: "Formatted at 2026-09-28 12:25:37 on dist-test-slave-920p"
server_key: "8f3840d45598f796cd6658e96f5817c9"
server_key_iv: "6070e3892846ac8ee1b2bc77497e7057"
server_key_version: "encryptionkey@0"
I20260928 12:25:37.437876 24394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "6b4b7e7660354b8995fe022491a42a9e"
format_stamp: "Formatted at 2026-09-28 12:25:37 on dist-test-slave-920p"
server_key: "8f3840d45598f796cd6658e96f5817c9"
server_key_iv: "6070e3892846ac8ee1b2bc77497e7057"
server_key_version: "encryptionkey@0"
I20260928 12:25:37.439040 24394 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:25:37.439687 24409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:37.439847 24394 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:25:37.439924 24394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "6b4b7e7660354b8995fe022491a42a9e"
format_stamp: "Formatted at 2026-09-28 12:25:37 on dist-test-slave-920p"
server_key: "8f3840d45598f796cd6658e96f5817c9"
server_key_iv: "6070e3892846ac8ee1b2bc77497e7057"
server_key_version: "encryptionkey@0"
I20260928 12:25:37.439975 24394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:37.443039 24394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:37.444955 24394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:37.445050 24394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:37.448467 24394 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:45739
I20260928 12:25:37.448518 24461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:45739 every 8 connection(s)
I20260928 12:25:37.448770 24394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:25:37.449764 24462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:37.451714 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 24394
I20260928 12:25:37.451797 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:25:37.451854 24462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b4b7e7660354b8995fe022491a42a9e: Bootstrap starting.
I20260928 12:25:37.451977 2272 external_mini_cluster.cc:1428] Setting key a5126afe7fb2ddbce74c72c345723de3
I20260928 12:25:37.452481 24462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b4b7e7660354b8995fe022491a42a9e: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:37.452783 24462 log.cc:824] T 00000000000000000000000000000000 P 6b4b7e7660354b8995fe022491a42a9e: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:37.453481 24462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b4b7e7660354b8995fe022491a42a9e: No bootstrap required, opened a new log
I20260928 12:25:37.454280 24465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:37.452402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35863 (local address 127.2.56.62:45739)
0928 12:25:37.452572 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:37.452581 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:37.452617 (+ 36us) server_negotiation.cc:407] Connection header received
0928 12:25:37.453150 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:37.453159 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:37.453191 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:37.453368 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:37.453878 (+ 510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:37.453882 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:37.453882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:37.453886 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:37.453927 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:37.453940 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:37.454033 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:37.454062 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 12:25:37.454098 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
I20260928 12:25:37.455109 2272 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:25:37.455623 24462 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b4b7e7660354b8995fe022491a42a9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4b7e7660354b8995fe022491a42a9e" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 45739 } }
I20260928 12:25:37.455745 24462 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b4b7e7660354b8995fe022491a42a9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:37.455765 24462 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b4b7e7660354b8995fe022491a42a9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b4b7e7660354b8995fe022491a42a9e, State: Initialized, Role: FOLLOWER
I20260928 12:25:37.455869 24462 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b4b7e7660354b8995fe022491a42a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4b7e7660354b8995fe022491a42a9e" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 45739 } }
I20260928 12:25:37.455950 24462 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b4b7e7660354b8995fe022491a42a9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:25:37.456015 24462 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b4b7e7660354b8995fe022491a42a9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:25:37.456068 24462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b4b7e7660354b8995fe022491a42a9e [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:37.456569 24462 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b4b7e7660354b8995fe022491a42a9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4b7e7660354b8995fe022491a42a9e" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 45739 } }
I20260928 12:25:37.456650 24462 leader_election.cc:302] T 00000000000000000000000000000000 P 6b4b7e7660354b8995fe022491a42a9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b4b7e7660354b8995fe022491a42a9e; no voters:
I20260928 12:25:37.456787 24462 leader_election.cc:288] T 00000000000000000000000000000000 P 6b4b7e7660354b8995fe022491a42a9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:25:37.456809 24471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b4b7e7660354b8995fe022491a42a9e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:25:37.456930 24471 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b4b7e7660354b8995fe022491a42a9e [term 1 LEADER]: Becoming Leader. State: Replica: 6b4b7e7660354b8995fe022491a42a9e, State: Running, Role: LEADER
I20260928 12:25:37.457000 24462 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b4b7e7660354b8995fe022491a42a9e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:25:37.457062 24471 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b4b7e7660354b8995fe022491a42a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4b7e7660354b8995fe022491a42a9e" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 45739 } }
I20260928 12:25:37.457412 24472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b4b7e7660354b8995fe022491a42a9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b4b7e7660354b8995fe022491a42a9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4b7e7660354b8995fe022491a42a9e" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 45739 } } }
I20260928 12:25:37.457497 24472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b4b7e7660354b8995fe022491a42a9e [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:37.457666 24473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b4b7e7660354b8995fe022491a42a9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b4b7e7660354b8995fe022491a42a9e. Latest consensus state: current_term: 1 leader_uuid: "6b4b7e7660354b8995fe022491a42a9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4b7e7660354b8995fe022491a42a9e" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 45739 } } }
I20260928 12:25:37.457751 24473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b4b7e7660354b8995fe022491a42a9e [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:37.457715 24479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:25:37.458240 24479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:25:37.459616 24465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:37.458380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36865 (local address 127.2.56.62:45739)
0928 12:25:37.458410 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:37.458412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:37.458736 (+ 324us) server_negotiation.cc:407] Connection header received
0928 12:25:37.458790 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:37.458791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:37.458796 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:37.458854 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:37.459388 (+ 534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:37.459389 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:37.459389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:25:37.459393 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:37.459407 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:25:37.459412 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:37.459482 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:37.459547 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 12:25:37.459560 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 12:25:37.465687 24479 catalog_manager.cc:1347] Generated new cluster ID: 346c0ab68b3b462a9f2a76bf2c75e07b
I20260928 12:25:37.465731 24479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:25:37.476294 24479 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:25:37.476450 24479 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:25:37.478868 24479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b4b7e7660354b8995fe022491a42a9e: Generated new TSK 0
I20260928 12:25:37.479017 24479 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:25:37.484841 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 24394
2026-09-28T12:25:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1197 ms)
[----------] 18 tests from AuthTokenIssuingTest (21964 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:25:37 dist-test-slave-920p krb5kdc[24493](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:25:37 dist-test-slave-920p krb5kdc[24493](info): set up 2 sockets
Sep 28 12:25:37 dist-test-slave-920p krb5kdc[24493](info): commencing operation
krb5kdc: starting...
W20260928 12:25:38.433223 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:25:38 dist-test-slave-920p krb5kdc[24493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:25:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:25:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
I20260928 12:25:38.541924 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:41815
--webserver_interface=127.2.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:42937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:41815,127.2.56.61:35687,127.2.56.60:34307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:25:38.622519 24506 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 12:25:38.622716 24506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:38.622731 24506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:38.622754 24506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:38.623893 24506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:25:38.623932 24506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:38.623951 24506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:38.623965 24506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:25:38.623987 24506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:25:38.625506 24506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:42937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:41815,127.2.56.61:35687,127.2.56.60:34307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:41815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:38.625726 24506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:38.625931 24506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:38.630556 24512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:38.630659 24506 server_base.cc:1034] running on GCE node
W20260928 12:25:38.630599 24511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:38.630604 24514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:38.631002 24506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:38.631294 24506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:38.632431 24506 hybrid_clock.cc:648] HybridClock initialized: now 1790598338632415 us; error 34 us; skew 500 ppm
Sep 28 12:25:38 dist-test-slave-920p krb5kdc[24493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598338, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:38.634727 24506 init.cc:377] Logged in from keytab as kudu/127.2.56.62@KRBTEST.COM (short username kudu)
I20260928 12:25:38.635085 24506 webserver.cc:460] Webserver started at http://127.2.56.62:44881/ using document root <none> and password file <none>
I20260928 12:25:38.635303 24506 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:38.635342 24506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:38.635434 24506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:38.636400 24506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "ea5d762193474930bd6ed7435a3dda5f"
format_stamp: "Formatted at 2026-09-28 12:25:38 on dist-test-slave-920p"
server_key: "4bdab7d810ec9713960e78127936c36e"
server_key_iv: "95a97a8de51a1b99e998c8f70056a86b"
server_key_version: "encryptionkey@0"
I20260928 12:25:38.636675 24506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "ea5d762193474930bd6ed7435a3dda5f"
format_stamp: "Formatted at 2026-09-28 12:25:38 on dist-test-slave-920p"
server_key: "4bdab7d810ec9713960e78127936c36e"
server_key_iv: "95a97a8de51a1b99e998c8f70056a86b"
server_key_version: "encryptionkey@0"
I20260928 12:25:38.637892 24506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:25:38.638597 24521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:38.638757 24506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:25:38.638833 24506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "ea5d762193474930bd6ed7435a3dda5f"
format_stamp: "Formatted at 2026-09-28 12:25:38 on dist-test-slave-920p"
server_key: "4bdab7d810ec9713960e78127936c36e"
server_key_iv: "95a97a8de51a1b99e998c8f70056a86b"
server_key_version: "encryptionkey@0"
I20260928 12:25:38.638888 24506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:38.674935 24506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:38.676909 24506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:38.677023 24506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:38.680547 24506 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:41815
I20260928 12:25:38.680591 24573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:41815 every 8 connection(s)
I20260928 12:25:38.680876 24506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:25:38.681841 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:38.683322 24574 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.56.62" port: 41815 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:25:38.686038 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 24506
I20260928 12:25:38.686149 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:25:38.686311 2272 external_mini_cluster.cc:1428] Setting key 61f09df23ac6bd39bc245238531ce944
Sep 28 12:25:38 dist-test-slave-920p krb5kdc[24493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598338, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:25:38.693102 24576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:38.683953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:41815 (local address 127.0.0.1:55738)
0928 12:25:38.684128 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:38.684168 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:38.684641 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:38.684894 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:38.684897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:38.685178 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:38.685352 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:38.685356 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:38.686595 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:38.686596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:38.687266 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:38.687268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:38.687760 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:38.687761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:38.687828 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:38.688336 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:25:38.688345 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:38.689860 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:38.691754 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:38.691760 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:38.691761 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:38.691937 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:38.692179 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:38.692180 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:38.692180 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:38.692246 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:38.692657 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:38.692660 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:38.692711 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 12:25:38.692774 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 12:25:38.692895 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":80,"threads_started":1}
I20260928 12:25:38.693109 24577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:38.684115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55738 (local address 127.2.56.62:41815)
0928 12:25:38.684238 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:38.684242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:38.684274 (+ 32us) server_negotiation.cc:407] Connection header received
0928 12:25:38.684783 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:38.684785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:38.684802 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:38.684876 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:38.685438 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:38.686509 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:38.687369 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:38.687674 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:38.687776 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:38.689959 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:38.689967 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:38.689968 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:38.689975 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:38.691681 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:38.692010 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:38.692013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:38.692014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:38.692115 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:38.692298 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:38.692298 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:38.692298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:38.692597 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:38.692682 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:38.692823 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 12:25:38.692971 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":100,"thread_start_us":42,"threads_started":1}
WARNING: no policy specified for kudu/127.2.56.61@KRBTEST.COM; defaulting to no policy
I20260928 12:25:38.693847 24574 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.56.61" port: 35687 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:25:38.694134 24576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:38.694043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.61:35687 (local address 127.0.0.1:35334)
0928 12:25:38.694074 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:38.694116 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.61:35687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260928 12:25:38.694262 24524 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.56.61:35687: connect: Connection refused (error 111)
I20260928 12:25:38.694621 24576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:38.694556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.61:35687 (local address 127.0.0.1:35348)
0928 12:25:38.694582 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:38.694605 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.61:35687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260928 12:25:38.694728 24574 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.56.61:35687: Network error: Client connection negotiation failed: client connection to 127.2.56.61:35687: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.61.keytab.
Entry for principal kudu/127.2.56.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.61.keytab.
I20260928 12:25:38.707881 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.61:35687
--webserver_interface=127.2.56.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.61.keytab
--principal=kudu/127.2.56.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:42937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:41815,127.2.56.61:35687,127.2.56.60:34307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:25:38.743916 24574 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.56.61" port: 35687 } attempt: 1
I20260928 12:25:38.744385 24576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:38.744261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.61:35687 (local address 127.0.0.1:35354)
0928 12:25:38.744316 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:38.744357 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.61:35687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 12:25:38.744683 24576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:38.744613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.61:35687 (local address 127.0.0.1:35366)
0928 12:25:38.744645 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:38.744667 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.61:35687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260928 12:25:38.744764 24574 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.56.61:35687: Network error: Client connection negotiation failed: client connection to 127.2.56.61:35687: connect: Connection refused (error 111)
W20260928 12:25:38.786389 24581 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 12:25:38.786576 24581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:38.786592 24581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:38.786609 24581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:38.787662 24581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:25:38.787695 24581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:38.787708 24581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:38.787719 24581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:25:38.787732 24581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:25:38.789129 24581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:42937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:41815,127.2.56.61:35687,127.2.56.60:34307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.61.keytab
--principal=kudu/127.2.56.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.61:35687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.56.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:38.789333 24581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:38.789507 24581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:38.793804 24587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:38.793872 24586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:38.793990 24589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:38.793977 24581 server_base.cc:1034] running on GCE node
I20260928 12:25:38.794217 24581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:38.794450 24581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:38.795588 24581 hybrid_clock.cc:648] HybridClock initialized: now 1790598338795570 us; error 27 us; skew 500 ppm
Sep 28 12:25:38 dist-test-slave-920p krb5kdc[24493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598338, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:38.797883 24581 init.cc:377] Logged in from keytab as kudu/127.2.56.61@KRBTEST.COM (short username kudu)
I20260928 12:25:38.798290 24581 webserver.cc:460] Webserver started at http://127.2.56.61:38145/ using document root <none> and password file <none>
I20260928 12:25:38.798487 24581 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:38.798533 24581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:38.798650 24581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:38.799626 24581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-1/data/instance:
uuid: "5e7db1b16bff417e864c4adfce440d5b"
format_stamp: "Formatted at 2026-09-28 12:25:38 on dist-test-slave-920p"
server_key: "f0c50d0da1f1789545c448957d0f4912"
server_key_iv: "5f16cb1431c5174c9dbcee096d4acf33"
server_key_version: "encryptionkey@0"
I20260928 12:25:38.799942 24581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-1/wal/instance:
uuid: "5e7db1b16bff417e864c4adfce440d5b"
format_stamp: "Formatted at 2026-09-28 12:25:38 on dist-test-slave-920p"
server_key: "f0c50d0da1f1789545c448957d0f4912"
server_key_iv: "5f16cb1431c5174c9dbcee096d4acf33"
server_key_version: "encryptionkey@0"
I20260928 12:25:38.801055 24581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 12:25:38.801739 24596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:38.801877 24581 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:38.801954 24581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-1/wal
uuid: "5e7db1b16bff417e864c4adfce440d5b"
format_stamp: "Formatted at 2026-09-28 12:25:38 on dist-test-slave-920p"
server_key: "f0c50d0da1f1789545c448957d0f4912"
server_key_iv: "5f16cb1431c5174c9dbcee096d4acf33"
server_key_version: "encryptionkey@0"
I20260928 12:25:38.802014 24581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:38.812965 24574 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.56.61" port: 35687 } attempt: 2
I20260928 12:25:38.813400 24576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:38.813297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.61:35687 (local address 127.0.0.1:35378)
0928 12:25:38.813339 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:38.813371 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.61:35687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 12:25:38.813711 24576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:38.813629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.61:35687 (local address 127.0.0.1:35394)
0928 12:25:38.813665 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:38.813689 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.61:35687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 12:25:38.813783 24574 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.56.61:35687: Network error: Client connection negotiation failed: client connection to 127.2.56.61:35687: connect: Connection refused (error 111)
I20260928 12:25:38.844610 24581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:38.846522 24581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:38.846662 24581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:38.850332 24581 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.61:35687
I20260928 12:25:38.850365 24648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.61:35687 every 8 connection(s)
I20260928 12:25:38.850687 24581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-1/data/info.pb
I20260928 12:25:38.851565 24649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:38.852293 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 24581
I20260928 12:25:38.852396 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-1/wal/instance
I20260928 12:25:38.852563 2272 external_mini_cluster.cc:1428] Setting key daef27278bdb52bf6fee62bf57256338
I20260928 12:25:38.853412 24649 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.56.62" port: 41815 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 12:25:38 dist-test-slave-920p krb5kdc[24493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598338, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.61@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.56.60@KRBTEST.COM; defaulting to no policy
I20260928 12:25:38.861528 24577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:38.854172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55744 (local address 127.2.56.62:41815)
0928 12:25:38.854221 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:38.854223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:38.854427 (+ 204us) server_negotiation.cc:407] Connection header received
0928 12:25:38.854899 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:38.854900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:38.854917 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:38.854967 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:38.855545 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:38.856084 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:38.856999 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:38.857202 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:38.857257 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:38.859240 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:38.859241 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:38.859242 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:38.859248 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:38.860292 (+ 1044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:38.860626 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:38.860627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:38.860628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:38.860679 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:38.860902 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:38.860903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:38.860903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:38.861005 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:38.861057 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:38.861354 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 12:25:38.861437 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.60@KRBTEST.COM" created.
I20260928 12:25:38.861748 24652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:38.854169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:41815 (local address 127.0.0.1:55744)
0928 12:25:38.854320 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:38.854354 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:38.854807 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:38.855032 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:38.855035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:38.855278 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:38.855461 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:38.855465 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:38.856171 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:38.856173 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:38.856922 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:38.856924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:38.857306 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:38.857307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:38.857414 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:38.857875 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:25:38.857882 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:38.859153 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:38.860359 (+ 1206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:38.860366 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:38.860368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:38.860565 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:38.860731 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:38.860732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:38.860732 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:38.860828 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:38.861089 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:38.861094 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:38.861213 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 12:25:38.861332 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 12:25:38.861511 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":69,"threads_started":1}
I20260928 12:25:38.862223 24649 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.56.61" port: 35687 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 12:25:38 dist-test-slave-920p krb5kdc[24493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598338, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.61@KRBTEST.COM for kudu/127.2.56.61@KRBTEST.COM
I20260928 12:25:38.868615 24652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:38.862439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.61:35687 (local address 127.0.0.1:35396)
0928 12:25:38.862477 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:38.862487 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:38.862521 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:38.862741 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:38.862742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:38.862873 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:38.862924 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:38.862926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:38.863734 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:38.863735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:38.864230 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:38.864232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:38.864665 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:38.864665 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:38.864723 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:38.865076 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:25:38.865079 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:38.866227 (+ 1148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:38.867682 (+ 1455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:38.867683 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:38.867684 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:38.867877 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:38.868043 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:38.868044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:38.868044 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:38.868105 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:38.868332 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:38.868333 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:38.868377 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 12:25:38.868451 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 12:25:38.868534 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 12:25:38.868680 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:38.862503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35396 (local address 127.2.56.61:35687)
0928 12:25:38.862624 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:38.862628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:38.862644 (+ 16us) server_negotiation.cc:407] Connection header received
0928 12:25:38.862662 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:38.862664 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:38.862674 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:38.862730 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:38.863023 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:38.863664 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:38.864297 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:38.864597 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:38.864653 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:38.866306 (+ 1653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:38.866312 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:38.866312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:38.866318 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:38.867602 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:38.867946 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:38.867948 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:38.867948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:38.867991 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:38.868164 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:38.868165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:38.868165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:38.868276 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:38.868329 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:38.868473 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 12:25:38.868577 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":95,"thread_start_us":73,"threads_started":1}
I20260928 12:25:38.869158 24649 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.56.60" port: 34307 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:25:38.869445 24652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:38.869353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:34307 (local address 127.0.0.1:36626)
0928 12:25:38.869385 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:38.869428 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.60:34307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260928 12:25:38.869565 24597 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.56.60:34307: connect: Connection refused (error 111)
I20260928 12:25:38.869897 24652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:38.869816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:34307 (local address 127.0.0.1:36636)
0928 12:25:38.869855 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:38.869879 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.60:34307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 12:25:38.869982 24649 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.56.60:34307: Network error: Client connection negotiation failed: client connection to 127.2.56.60:34307: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.60.keytab.
Entry for principal kudu/127.2.56.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.60.keytab.
I20260928 12:25:38.876441 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.60:34307
--webserver_interface=127.2.56.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.60.keytab
--principal=kudu/127.2.56.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:42937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:41815,127.2.56.61:35687,127.2.56.60:34307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:25:38.904927 24574 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.56.61" port: 35687 } attempt: 3
Sep 28 12:25:38 dist-test-slave-920p krb5kdc[24493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598338, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for kudu/127.2.56.61@KRBTEST.COM
I20260928 12:25:38.911541 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:38.905357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35404 (local address 127.2.56.61:35687)
0928 12:25:38.905406 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:38.905408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:38.905451 (+ 43us) server_negotiation.cc:407] Connection header received
0928 12:25:38.905534 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:38.905534 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:38.905546 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:38.905594 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:38.905905 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:38.906416 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:38.907072 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:38.907293 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:38.907347 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:38.909295 (+ 1948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:38.909296 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:38.909297 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:38.909304 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:38.910538 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:38.910844 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:38.910845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:38.910846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:38.910894 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:38.911071 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:38.911072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:38.911072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:38.911176 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:38.911229 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:38.911360 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 12:25:38.911431 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 12:25:38.911541 24576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:38.905337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.61:35687 (local address 127.0.0.1:35404)
0928 12:25:38.905385 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:38.905399 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:38.905471 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:38.905617 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:38.905618 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:38.905766 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:38.905830 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:38.905833 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:38.906489 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:38.906490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:38.906998 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:38.906999 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:38.907351 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:38.907352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:38.907423 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:38.907867 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:25:38.907871 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:38.909194 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:38.910625 (+ 1431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:38.910626 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:38.910627 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:38.910783 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:38.910947 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:38.910947 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:38.910948 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:38.911002 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:38.911229 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:38.911230 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:38.911275 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 12:25:38.911335 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 12:25:38.911430 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 12:25:38.911936 24574 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.56.60" port: 34307 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:25:38.912230 24576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:38.912143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:34307 (local address 127.0.0.1:36642)
0928 12:25:38.912182 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:38.912207 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.60:34307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 12:25:38.912475 24576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:38.912400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:34307 (local address 127.0.0.1:36656)
0928 12:25:38.912433 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:38.912454 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.60:34307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 12:25:38.912566 24574 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.56.60:34307: Network error: Client connection negotiation failed: client connection to 127.2.56.60:34307: connect: Connection refused (error 111)
I20260928 12:25:38.919185 24649 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.56.60" port: 34307 } attempt: 1
I20260928 12:25:38.919663 24652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:38.919484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:34307 (local address 127.0.0.1:36660)
0928 12:25:38.919593 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:38.919638 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.60:34307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260928 12:25:38.920037 24652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:38.919907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:34307 (local address 127.0.0.1:36666)
0928 12:25:38.919981 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:38.920012 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.60:34307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 12:25:38.920130 24649 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.56.60:34307: Network error: Client connection negotiation failed: client connection to 127.2.56.60:34307: connect: Connection refused (error 111)
I20260928 12:25:38.943743 24574 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.56.60" port: 34307 } attempt: 1
I20260928 12:25:38.944238 24576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:38.944092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:34307 (local address 127.0.0.1:36678)
0928 12:25:38.944155 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:38.944200 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.60:34307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 12:25:38.944588 24576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:38.944482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:34307 (local address 127.0.0.1:36694)
0928 12:25:38.944524 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:38.944563 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.60:34307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 12:25:38.944675 24574 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.56.60:34307: Network error: Client connection negotiation failed: client connection to 127.2.56.60:34307: connect: Connection refused (error 111)
W20260928 12:25:38.955587 24656 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 12:25:38.955778 24656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:38.955793 24656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:38.955812 24656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:38.956913 24656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:25:38.956956 24656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:38.956970 24656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:38.956995 24656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:25:38.957015 24656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:25:38.958499 24656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:42937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:41815,127.2.56.61:35687,127.2.56.60:34307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.60.keytab
--principal=kudu/127.2.56.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.60:34307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.56.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:38.958797 24656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:38.959031 24656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:38.963537 24661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:38.963559 24664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:38.963559 24662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:38.963798 24656 server_base.cc:1034] running on GCE node
I20260928 12:25:38.964026 24656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:38.964278 24656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:38.965416 24656 hybrid_clock.cc:648] HybridClock initialized: now 1790598338965405 us; error 25 us; skew 500 ppm
Sep 28 12:25:38 dist-test-slave-920p krb5kdc[24493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598338, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:38.967613 24656 init.cc:377] Logged in from keytab as kudu/127.2.56.60@KRBTEST.COM (short username kudu)
I20260928 12:25:38.967993 24656 webserver.cc:460] Webserver started at http://127.2.56.60:33323/ using document root <none> and password file <none>
I20260928 12:25:38.968184 24656 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:38.968230 24656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:38.968348 24656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:38.969358 24656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-2/data/instance:
uuid: "e15d437594de481cae848da2e2e2b820"
format_stamp: "Formatted at 2026-09-28 12:25:38 on dist-test-slave-920p"
server_key: "717f385213fc9a738a4382d93fd4074f"
server_key_iv: "3fa51609aa58fd7d3aac7647ee5de015"
server_key_version: "encryptionkey@0"
I20260928 12:25:38.969676 24656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-2/wal/instance:
uuid: "e15d437594de481cae848da2e2e2b820"
format_stamp: "Formatted at 2026-09-28 12:25:38 on dist-test-slave-920p"
server_key: "717f385213fc9a738a4382d93fd4074f"
server_key_iv: "3fa51609aa58fd7d3aac7647ee5de015"
server_key_version: "encryptionkey@0"
I20260928 12:25:38.970839 24656 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:25:38.971506 24671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:38.971686 24656 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:25:38.971762 24656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-2/wal
uuid: "e15d437594de481cae848da2e2e2b820"
format_stamp: "Formatted at 2026-09-28 12:25:38 on dist-test-slave-920p"
server_key: "717f385213fc9a738a4382d93fd4074f"
server_key_iv: "3fa51609aa58fd7d3aac7647ee5de015"
server_key_version: "encryptionkey@0"
I20260928 12:25:38.971827 24656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:38.986701 24656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:38.988323 24656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:38.988274 24649 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.56.60" port: 34307 } attempt: 2
I20260928 12:25:38.988456 24656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:38.988644 24652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:38.988514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:34307 (local address 127.0.0.1:36708)
0928 12:25:38.988570 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:38.988608 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.60:34307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 12:25:38.988950 24652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:38.988867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:34307 (local address 127.0.0.1:36714)
0928 12:25:38.988907 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:38.988931 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.60:34307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 12:25:38.989034 24649 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.56.60:34307: Network error: Client connection negotiation failed: client connection to 127.2.56.60:34307: connect: Connection refused (error 111)
I20260928 12:25:38.992379 24656 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.60:34307
I20260928 12:25:38.992417 24723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.60:34307 every 8 connection(s)
I20260928 12:25:38.992769 24656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-2/data/info.pb
I20260928 12:25:38.993930 24724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:38.995966 24724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.56.62" port: 41815 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:25:39.000655 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 24656
I20260928 12:25:39.000733 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598072638673-2272-0/minicluster-data/master-2/wal/instance
I20260928 12:25:39.000877 2272 external_mini_cluster.cc:1428] Setting key 5b55127839d6b059a069a8f315fe2d65
Sep 28 12:25:38 dist-test-slave-920p krb5kdc[24493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598338, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.60@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:25:39.004523 24577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:38.996955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55752 (local address 127.2.56.62:41815)
0928 12:25:38.997350 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:38.997352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:38.997360 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:25:38.997405 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:38.997406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:38.997417 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:38.997463 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:38.998218 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:38.998688 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:38.999594 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:38.999788 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:38.999842 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:39.002005 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:39.002007 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:39.002007 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:39.002015 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:39.003330 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:39.003670 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:39.003671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:39.003671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:39.003727 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:39.003908 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:39.003909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:39.003909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:39.004034 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:39.004088 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:39.004358 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 12:25:39.004435 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365}
I20260928 12:25:39.004662 24726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:38.996614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:41815 (local address 127.0.0.1:55752)
0928 12:25:38.996756 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:38.996794 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:38.997290 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:38.997540 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:38.997544 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:38.997893 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:38.998131 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:38.998136 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:38.998763 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:38.998764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:38.999519 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:38.999521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:38.999869 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:38.999870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:38.999987 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:39.000426 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:25:39.000434 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:39.001875 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:39.003393 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:39.003396 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:39.003398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:39.003606 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:39.003780 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:39.003780 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:39.003781 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:39.003861 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:39.004107 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:39.004113 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:39.004242 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 12:25:39.004330 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 12:25:39.004461 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":70,"threads_started":1}
I20260928 12:25:39.005223 24724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.56.61" port: 35687 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 12:25:39 dist-test-slave-920p krb5kdc[24493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.60@KRBTEST.COM
Sep 28 12:25:39 dist-test-slave-920p krb5kdc[24493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598338, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.60@KRBTEST.COM for kudu/127.2.56.61@KRBTEST.COM
I20260928 12:25:39.009658 24728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:39.001226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36726 (local address 127.2.56.60:34307)
0928 12:25:39.001398 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:39.001402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:39.001425 (+ 23us) server_negotiation.cc:407] Connection header received
0928 12:25:39.001475 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:39.001478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:39.001497 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:39.001555 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:39.002218 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:39.003074 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:39.003958 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:39.004274 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:39.004363 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:39.006953 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:39.006961 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:39.006961 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:39.006967 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:39.008408 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:39.008729 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:39.008731 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:39.008731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:39.008780 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:39.008947 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:39.008947 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:39.008948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:39.009096 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:39.009212 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:39.009482 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 12:25:39.009551 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":115,"threads_started":1}
I20260928 12:25:39.011629 24726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:39.005501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.61:35687 (local address 127.0.0.1:35410)
0928 12:25:39.005534 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:39.005544 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:39.005582 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:39.005700 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:39.005700 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:39.005797 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:39.005837 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:39.005838 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:39.006489 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:39.006490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:39.007030 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:39.007032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:39.007431 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:39.007432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:39.007481 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:39.007768 (+ 287us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:25:39.007772 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:39.008837 (+ 1065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:39.010809 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:39.010810 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:39.010811 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:39.010994 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:39.011168 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:39.011168 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:39.011169 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:39.011212 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:39.011428 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:39.011429 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:39.011458 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 12:25:39.011513 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 12:25:39.011573 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 12:25:39.011691 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:39.005534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35410 (local address 127.2.56.61:35687)
0928 12:25:39.005585 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:39.005587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:39.005598 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:25:39.005627 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:39.005628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:39.005639 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:39.005698 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:39.005892 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:39.006424 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:39.007100 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:39.007365 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:39.007427 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:39.008907 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:39.008908 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:39.008908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:39.008914 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:39.010745 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:39.011047 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:39.011048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:39.011048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:39.011104 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:39.011256 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:39.011256 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:39.011257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:39.011378 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:39.011435 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:39.011543 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 12:25:39.011612 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:25:39.011966 24724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.56.60" port: 34307 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 12:25:39 dist-test-slave-920p krb5kdc[24493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
Sep 28 12:25:39 dist-test-slave-920p krb5kdc[24493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598338, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.60@KRBTEST.COM for kudu/127.2.56.60@KRBTEST.COM
I20260928 12:25:39.017143 24577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:39.010446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55762 (local address 127.2.56.62:41815)
0928 12:25:39.010485 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:39.010487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:39.010495 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:25:39.010516 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:39.010516 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:39.010525 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:39.010584 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:39.011035 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:39.011671 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:39.012506 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:39.012752 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:39.012815 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:39.015041 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:39.015042 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:39.015043 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:39.015048 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:39.016177 (+ 1129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:39.016500 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:39.016501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:39.016502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:39.016556 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:39.016709 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:39.016709 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:39.016710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:39.016810 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:39.016946 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:39.016990 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 12:25:39.017060 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:25:39.018118 24726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:39.012178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:34307 (local address 127.0.0.1:36732)
0928 12:25:39.012231 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:39.012239 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:39.012274 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:39.012373 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:39.012374 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:39.012497 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:39.012527 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:39.012528 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:39.013261 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:39.013287 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:39.014009 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:39.014012 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:39.014429 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:39.014430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:39.014465 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:39.014776 (+ 311us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:25:39.014781 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:39.015931 (+ 1150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:39.017220 (+ 1289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:39.017221 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:39.017221 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:39.017402 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:39.017545 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:39.017546 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:39.017546 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:39.017594 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:39.017830 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:39.017831 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:39.017868 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 12:25:39.017932 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 12:25:39.018031 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 12:25:39.018292 24728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:39.012202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36732 (local address 127.2.56.60:34307)
0928 12:25:39.012230 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:39.012231 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:39.012263 (+ 32us) server_negotiation.cc:407] Connection header received
0928 12:25:39.012319 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:39.012320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:39.012326 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:39.012362 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:39.012581 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:39.013193 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:39.014082 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:39.014366 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:39.014432 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:39.016023 (+ 1591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:39.016025 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:39.016025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:39.016032 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:39.017154 (+ 1122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:39.017456 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:39.017457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:39.017457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:39.017498 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:39.017642 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:39.017643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:39.017643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:39.017770 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:39.017830 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:39.017954 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 12:25:39.018206 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":170,"server-negotiator.queue_time_us":13}
I20260928 12:25:39.019585 24724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e15d437594de481cae848da2e2e2b820: Bootstrap starting.
I20260928 12:25:39.019791 24574 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.56.60" port: 34307 } attempt: 2
I20260928 12:25:39.020467 24724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e15d437594de481cae848da2e2e2b820: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:39.020758 24724 log.cc:824] T 00000000000000000000000000000000 P e15d437594de481cae848da2e2e2b820: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:39.021562 24724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e15d437594de481cae848da2e2e2b820: No bootstrap required, opened a new log
Sep 28 12:25:39 dist-test-slave-920p krb5kdc[24493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.61@KRBTEST.COM
Sep 28 12:25:39 dist-test-slave-920p krb5kdc[24493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598338, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for kudu/127.2.56.60@KRBTEST.COM
I20260928 12:25:39.023613 24724 raft_consensus.cc:348] T 00000000000000000000000000000000 P e15d437594de481cae848da2e2e2b820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5d762193474930bd6ed7435a3dda5f" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 41815 } } peers { permanent_uuid: "5e7db1b16bff417e864c4adfce440d5b" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 35687 } } peers { permanent_uuid: "e15d437594de481cae848da2e2e2b820" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 34307 } }
I20260928 12:25:39.023761 24724 raft_consensus.cc:729] T 00000000000000000000000000000000 P e15d437594de481cae848da2e2e2b820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e15d437594de481cae848da2e2e2b820, State: Initialized, Role: FOLLOWER
I20260928 12:25:39.023845 24724 consensus_queue.cc:260] T 00000000000000000000000000000000 P e15d437594de481cae848da2e2e2b820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5d762193474930bd6ed7435a3dda5f" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 41815 } } peers { permanent_uuid: "5e7db1b16bff417e864c4adfce440d5b" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 35687 } } peers { permanent_uuid: "e15d437594de481cae848da2e2e2b820" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 34307 } }
I20260928 12:25:39.024144 24724 sys_catalog.cc:564] T 00000000000000000000000000000000 P e15d437594de481cae848da2e2e2b820 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:25:39.025223 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:39.018017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35412 (local address 127.2.56.61:35687)
0928 12:25:39.018051 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:39.018052 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:39.018058 (+ 6us) server_negotiation.cc:407] Connection header received
0928 12:25:39.018076 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:39.018077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:39.018084 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:39.018124 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:39.018674 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:39.019091 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:39.020016 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:39.020290 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:39.020350 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:39.022634 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:39.022635 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:39.022635 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:39.022642 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:39.024183 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:39.024535 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:39.024536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:39.024537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:39.024593 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:39.024762 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:39.024763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:39.024764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:39.024883 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:39.025048 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:39.025111 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 12:25:39.025167 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:25:39.026611 24576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:39.020020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:34307 (local address 127.0.0.1:36740)
0928 12:25:39.020091 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:39.020100 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:39.020133 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:39.020361 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:39.020362 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:39.020502 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:39.020539 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:39.020540 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:39.021288 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:39.021289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:39.021832 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:39.021834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:39.022147 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:39.022149 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:39.022208 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:39.022537 (+ 329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:25:39.022542 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:39.023964 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:39.025537 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:39.025538 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:39.025539 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:39.025697 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:39.025879 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:39.025880 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:39.025881 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:39.025955 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:39.026291 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:39.026292 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:39.026338 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 12:25:39.026403 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 12:25:39.026524 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 12:25:39.026853 24728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:39.020195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36740 (local address 127.2.56.60:34307)
0928 12:25:39.020252 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:39.020254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:39.020260 (+ 6us) server_negotiation.cc:407] Connection header received
0928 12:25:39.020277 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:39.020277 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:39.020285 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:39.020604 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:39.020631 (+ 27us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:39.021209 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:39.021899 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:39.022094 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:39.022144 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:39.024059 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:39.024060 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:39.024061 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:39.024067 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:39.025473 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:39.025768 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:39.025769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:39.025770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:39.025824 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:39.026024 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:39.026025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:39.026025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:39.026177 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:39.026260 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:39.026716 (+ 456us) server_negotiation.cc:299] Negotiation successful
0928 12:25:39.026784 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260928 12:25:39.026984 24741 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e15d437594de481cae848da2e2e2b820: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:25:39.027016 24741 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:25:39.026439 24730 sys_catalog.cc:455] T 00000000000000000000000000000000 P e15d437594de481cae848da2e2e2b820 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5d762193474930bd6ed7435a3dda5f" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 41815 } } peers { permanent_uuid: "5e7db1b16bff417e864c4adfce440d5b" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 35687 } } peers { permanent_uuid: "e15d437594de481cae848da2e2e2b820" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 34307 } } }
I20260928 12:25:39.027277 24730 sys_catalog.cc:458] T 00000000000000000000000000000000 P e15d437594de481cae848da2e2e2b820 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:25:39.028191 24574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea5d762193474930bd6ed7435a3dda5f: Bootstrap starting.
I20260928 12:25:39.028858 24574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea5d762193474930bd6ed7435a3dda5f: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:39.029153 24574 log.cc:824] T 00000000000000000000000000000000 P ea5d762193474930bd6ed7435a3dda5f: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:39.029376 2272 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:25:39.027395 24656 master_runner.cc:186] Error getting master registration for 127.2.56.62:41815: OK, instance_id { permanent_uuid: "ea5d762193474930bd6ed7435a3dda5f" instance_seqno: 1790598338679595 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 12:25:39.029558 24656 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 12:25:39.030269 24548 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ea5d762193474930bd6ed7435a3dda5f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55762
I20260928 12:25:39.036139 24574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea5d762193474930bd6ed7435a3dda5f: No bootstrap required, opened a new log
I20260928 12:25:39.037459 24574 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea5d762193474930bd6ed7435a3dda5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5d762193474930bd6ed7435a3dda5f" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 41815 } } peers { permanent_uuid: "5e7db1b16bff417e864c4adfce440d5b" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 35687 } } peers { permanent_uuid: "e15d437594de481cae848da2e2e2b820" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 34307 } }
I20260928 12:25:39.037595 24574 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea5d762193474930bd6ed7435a3dda5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea5d762193474930bd6ed7435a3dda5f, State: Initialized, Role: FOLLOWER
I20260928 12:25:39.037674 24574 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea5d762193474930bd6ed7435a3dda5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5d762193474930bd6ed7435a3dda5f" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 41815 } } peers { permanent_uuid: "5e7db1b16bff417e864c4adfce440d5b" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 35687 } } peers { permanent_uuid: "e15d437594de481cae848da2e2e2b820" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 34307 } }
I20260928 12:25:39.037971 24574 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea5d762193474930bd6ed7435a3dda5f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:25:39.037927 24744 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea5d762193474930bd6ed7435a3dda5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5d762193474930bd6ed7435a3dda5f" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 41815 } } peers { permanent_uuid: "5e7db1b16bff417e864c4adfce440d5b" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 35687 } } peers { permanent_uuid: "e15d437594de481cae848da2e2e2b820" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 34307 } } }
I20260928 12:25:39.038021 24548 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea5d762193474930bd6ed7435a3dda5f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 12:25:39.038035 24744 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea5d762193474930bd6ed7435a3dda5f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:25:39.038074 24548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea5d762193474930bd6ed7435a3dda5f [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:39.038698 24548 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea5d762193474930bd6ed7435a3dda5f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5d762193474930bd6ed7435a3dda5f" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 41815 } } peers { permanent_uuid: "5e7db1b16bff417e864c4adfce440d5b" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 35687 } } peers { permanent_uuid: "e15d437594de481cae848da2e2e2b820" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 34307 } }
I20260928 12:25:39.038879 24548 leader_election.cc:288] T 00000000000000000000000000000000 P ea5d762193474930bd6ed7435a3dda5f [CANDIDATE]: Term 1 election: Requested vote from peers 5e7db1b16bff417e864c4adfce440d5b (127.2.56.61:35687), e15d437594de481cae848da2e2e2b820 (127.2.56.60:34307)
I20260928 12:25:39.039139 24698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ea5d762193474930bd6ed7435a3dda5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e15d437594de481cae848da2e2e2b820"
I20260928 12:25:39.039251 24698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e15d437594de481cae848da2e2e2b820 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:39.040076 24698 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e15d437594de481cae848da2e2e2b820 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea5d762193474930bd6ed7435a3dda5f in term 1.
I20260928 12:25:39.040292 24522 leader_election.cc:302] T 00000000000000000000000000000000 P ea5d762193474930bd6ed7435a3dda5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e15d437594de481cae848da2e2e2b820, ea5d762193474930bd6ed7435a3dda5f; no voters:
I20260928 12:25:39.040417 24744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea5d762193474930bd6ed7435a3dda5f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:25:39.040548 24744 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea5d762193474930bd6ed7435a3dda5f [term 1 LEADER]: Becoming Leader. State: Replica: ea5d762193474930bd6ed7435a3dda5f, State: Running, Role: LEADER
I20260928 12:25:39.040630 24744 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea5d762193474930bd6ed7435a3dda5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5d762193474930bd6ed7435a3dda5f" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 41815 } } peers { permanent_uuid: "5e7db1b16bff417e864c4adfce440d5b" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 35687 } } peers { permanent_uuid: "e15d437594de481cae848da2e2e2b820" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 34307 } }
I20260928 12:25:39.041034 24759 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea5d762193474930bd6ed7435a3dda5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea5d762193474930bd6ed7435a3dda5f. Latest consensus state: current_term: 1 leader_uuid: "ea5d762193474930bd6ed7435a3dda5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5d762193474930bd6ed7435a3dda5f" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 41815 } } peers { permanent_uuid: "5e7db1b16bff417e864c4adfce440d5b" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 35687 } } peers { permanent_uuid: "e15d437594de481cae848da2e2e2b820" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 34307 } } }
I20260928 12:25:39.041225 24759 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea5d762193474930bd6ed7435a3dda5f [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:39.043188 24506 master_runner.cc:186] Error getting master registration for 127.2.56.61:35687: OK, instance_id { permanent_uuid: "5e7db1b16bff417e864c4adfce440d5b" instance_seqno: 1790598338849254 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 12:25:39.045455 24506 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 12:25:39.045806 24765 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ea5d762193474930bd6ed7435a3dda5f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:25:39.045852 24765 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:25:39.048179 24577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:39.040230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55770 (local address 127.2.56.62:41815)
0928 12:25:39.040272 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:39.040273 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:39.040500 (+ 227us) server_negotiation.cc:407] Connection header received
0928 12:25:39.040591 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:39.040592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:39.040601 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:39.040629 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:39.041395 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:39.042070 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:39.043112 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:39.043381 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:39.043428 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:39.044790 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:39.044791 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:39.044792 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:39.044798 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:39.047041 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:39.047390 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:39.047391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:39.047392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:39.047447 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:39.047636 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:39.047636 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:39.047637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:39.047751 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:39.047820 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:39.048019 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 12:25:39.048092 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 12:25:39.048609 24728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:39.040229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36748 (local address 127.2.56.60:34307)
0928 12:25:39.041013 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:39.041015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:39.041951 (+ 936us) server_negotiation.cc:407] Connection header received
0928 12:25:39.042030 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:39.042031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:39.042040 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:39.042066 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:39.042667 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:39.043354 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:39.044347 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:39.044643 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:39.044680 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:39.046066 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:39.046067 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:39.046067 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:39.046073 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:39.047483 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:39.047844 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:39.047845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:39.047846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:39.047896 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:39.048081 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:39.048082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:39.048082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:39.048194 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:39.048255 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:39.048450 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 12:25:39.048522 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":765}
I20260928 12:25:39.049595 24762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:25:39.049787 24762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:25:39.051272 24698 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e15d437594de481cae848da2e2e2b820 [term 1 FOLLOWER]: Refusing update from remote peer ea5d762193474930bd6ed7435a3dda5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:25:39.051573 24744 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ea5d762193474930bd6ed7435a3dda5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e15d437594de481cae848da2e2e2b820" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 34307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:25:39.053385 24730 sys_catalog.cc:455] T 00000000000000000000000000000000 P e15d437594de481cae848da2e2e2b820 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea5d762193474930bd6ed7435a3dda5f. Latest consensus state: current_term: 1 leader_uuid: "ea5d762193474930bd6ed7435a3dda5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5d762193474930bd6ed7435a3dda5f" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 41815 } } peers { permanent_uuid: "5e7db1b16bff417e864c4adfce440d5b" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 35687 } } peers { permanent_uuid: "e15d437594de481cae848da2e2e2b820" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 34307 } } }
I20260928 12:25:39.053468 24730 sys_catalog.cc:458] T 00000000000000000000000000000000 P e15d437594de481cae848da2e2e2b820 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:25:39.055378 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:39.040007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35416 (local address 127.2.56.61:35687)
0928 12:25:39.040042 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:39.040044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:39.041783 (+ 1739us) server_negotiation.cc:407] Connection header received
0928 12:25:39.041843 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:39.041844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:39.041868 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:39.041917 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:39.046548 (+ 4631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:39.047221 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:39.049059 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:39.049392 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:39.049436 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:39.052284 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:39.052296 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:39.052297 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:39.052304 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:39.054265 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:39.054614 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:39.054615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:39.054615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:39.054671 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:39.054866 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:39.054867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:39.054868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:39.054997 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:39.055084 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:39.055206 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 12:25:39.055292 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 12:25:39.056228 24759 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea5d762193474930bd6ed7435a3dda5f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ea5d762193474930bd6ed7435a3dda5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5d762193474930bd6ed7435a3dda5f" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 41815 } } peers { permanent_uuid: "5e7db1b16bff417e864c4adfce440d5b" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 35687 } } peers { permanent_uuid: "e15d437594de481cae848da2e2e2b820" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 34307 } } }
I20260928 12:25:39.056305 24759 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea5d762193474930bd6ed7435a3dda5f [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:39.056332 24730 sys_catalog.cc:455] T 00000000000000000000000000000000 P e15d437594de481cae848da2e2e2b820 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ea5d762193474930bd6ed7435a3dda5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5d762193474930bd6ed7435a3dda5f" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 41815 } } peers { permanent_uuid: "5e7db1b16bff417e864c4adfce440d5b" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 35687 } } peers { permanent_uuid: "e15d437594de481cae848da2e2e2b820" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 34307 } } }
I20260928 12:25:39.056391 24730 sys_catalog.cc:458] T 00000000000000000000000000000000 P e15d437594de481cae848da2e2e2b820 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:25:39.056541 24767 mvcc.cc:205] Tried to move back new op lower bound from 7334290796751466496 to 7334290796710981632. Current Snapshot: MvccSnapshot[applied={T|T < 7334290796751466496}]
I20260928 12:25:39.057348 24762 catalog_manager.cc:1347] Generated new cluster ID: f12cc0ff973a41279a8fc26d8414c204
I20260928 12:25:39.057391 24762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:25:39.057647 24768 mvcc.cc:205] Tried to move back new op lower bound from 7334290796751466496 to 7334290796710981632. Current Snapshot: MvccSnapshot[applied={T|T < 7334290796751466496}]
I20260928 12:25:39.080156 24649 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.56.60" port: 34307 } attempt: 3
I20260928 12:25:39.083287 24762 catalog_manager.cc:1370] Generated new certificate authority record
Sep 28 12:25:39 dist-test-slave-920p krb5kdc[24493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598338, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.61@KRBTEST.COM for kudu/127.2.56.60@KRBTEST.COM
I20260928 12:25:39.084744 24762 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:25:39.087782 24652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:39.080457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:34307 (local address 127.0.0.1:36752)
0928 12:25:39.080542 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:39.080554 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:39.080595 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:39.080870 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:39.080871 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:39.081060 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:39.081107 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:39.081109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:39.081858 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:39.081859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:39.082540 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:39.082554 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:39.083131 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:39.083132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:39.083189 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:39.083700 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:25:39.083706 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:39.085115 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:39.086842 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:39.086843 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:39.086843 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:39.087031 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:39.087205 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:39.087206 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:39.087206 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:39.087258 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:39.087490 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:39.087491 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:39.087530 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 12:25:39.087596 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 12:25:39.087686 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 12:25:39.088076 24728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:39.080708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36752 (local address 127.2.56.60:34307)
0928 12:25:39.080746 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:39.080747 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:39.080754 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:25:39.080774 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:39.080775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:39.080783 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:39.080822 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:39.081182 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:39.081772 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:39.082737 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:39.082993 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:39.083047 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:39.085193 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:39.085194 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:39.085195 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:39.085201 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:39.086764 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:39.087095 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:39.087096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:39.087097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:39.087148 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:39.087312 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:39.087312 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:39.087312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:39.087422 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:39.087469 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:39.087931 (+ 462us) server_negotiation.cc:299] Negotiation successful
0928 12:25:39.087995 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:25:39.089475 24649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e7db1b16bff417e864c4adfce440d5b: Bootstrap starting.
I20260928 12:25:39.090164 24649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e7db1b16bff417e864c4adfce440d5b: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:39.090453 24649 log.cc:824] T 00000000000000000000000000000000 P 5e7db1b16bff417e864c4adfce440d5b: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:39.102042 24649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e7db1b16bff417e864c4adfce440d5b: No bootstrap required, opened a new log
I20260928 12:25:39.104992 24649 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e7db1b16bff417e864c4adfce440d5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5d762193474930bd6ed7435a3dda5f" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 41815 } } peers { permanent_uuid: "5e7db1b16bff417e864c4adfce440d5b" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 35687 } } peers { permanent_uuid: "e15d437594de481cae848da2e2e2b820" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 34307 } }
I20260928 12:25:39.105211 24649 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e7db1b16bff417e864c4adfce440d5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e7db1b16bff417e864c4adfce440d5b, State: Initialized, Role: FOLLOWER
I20260928 12:25:39.105399 24649 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e7db1b16bff417e864c4adfce440d5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5d762193474930bd6ed7435a3dda5f" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 41815 } } peers { permanent_uuid: "5e7db1b16bff417e864c4adfce440d5b" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 35687 } } peers { permanent_uuid: "e15d437594de481cae848da2e2e2b820" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 34307 } }
I20260928 12:25:39.106042 24623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ea5d762193474930bd6ed7435a3dda5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5e7db1b16bff417e864c4adfce440d5b"
I20260928 12:25:39.106137 24623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e7db1b16bff417e864c4adfce440d5b [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:39.106981 24623 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5e7db1b16bff417e864c4adfce440d5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea5d762193474930bd6ed7435a3dda5f in term 1.
I20260928 12:25:39.107345 24622 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5e7db1b16bff417e864c4adfce440d5b [term 1 FOLLOWER]: Refusing update from remote peer ea5d762193474930bd6ed7435a3dda5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:25:39.107674 24744 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ea5d762193474930bd6ed7435a3dda5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e7db1b16bff417e864c4adfce440d5b" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 35687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:25:39.107954 24649 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e7db1b16bff417e864c4adfce440d5b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:25:39.109414 24762 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea5d762193474930bd6ed7435a3dda5f: Generated new TSK 0
I20260928 12:25:39.109629 24762 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:25:39.110041 24787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e7db1b16bff417e864c4adfce440d5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea5d762193474930bd6ed7435a3dda5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5d762193474930bd6ed7435a3dda5f" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 41815 } } peers { permanent_uuid: "5e7db1b16bff417e864c4adfce440d5b" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 35687 } } peers { permanent_uuid: "e15d437594de481cae848da2e2e2b820" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 34307 } } }
I20260928 12:25:39.110214 24787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e7db1b16bff417e864c4adfce440d5b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:25:39.110746 24787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e7db1b16bff417e864c4adfce440d5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea5d762193474930bd6ed7435a3dda5f. Latest consensus state: current_term: 1 leader_uuid: "ea5d762193474930bd6ed7435a3dda5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5d762193474930bd6ed7435a3dda5f" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 41815 } } peers { permanent_uuid: "5e7db1b16bff417e864c4adfce440d5b" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 35687 } } peers { permanent_uuid: "e15d437594de481cae848da2e2e2b820" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 34307 } } }
I20260928 12:25:39.110877 24787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e7db1b16bff417e864c4adfce440d5b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:25:39.110965 24787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e7db1b16bff417e864c4adfce440d5b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ea5d762193474930bd6ed7435a3dda5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5d762193474930bd6ed7435a3dda5f" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 41815 } } peers { permanent_uuid: "5e7db1b16bff417e864c4adfce440d5b" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 35687 } } peers { permanent_uuid: "e15d437594de481cae848da2e2e2b820" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 34307 } } }
I20260928 12:25:39.111061 24787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e7db1b16bff417e864c4adfce440d5b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:25:39.111629 24780 mvcc.cc:205] Tried to move back new op lower bound from 7334290796879482880 to 7334290796710981632. Current Snapshot: MvccSnapshot[applied={T|T < 7334290796751466496}]
I20260928 12:25:39.111948 24759 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea5d762193474930bd6ed7435a3dda5f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ea5d762193474930bd6ed7435a3dda5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5d762193474930bd6ed7435a3dda5f" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 41815 } } peers { permanent_uuid: "5e7db1b16bff417e864c4adfce440d5b" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 35687 } } peers { permanent_uuid: "e15d437594de481cae848da2e2e2b820" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 34307 } } }
I20260928 12:25:39.112033 24759 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea5d762193474930bd6ed7435a3dda5f [sys.catalog]: This master's current role is: LEADER
W20260928 12:25:39.113881 24788 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e7db1b16bff417e864c4adfce440d5b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:25:39.113988 24788 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260928 12:25:39.131254 24653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:25:39.129845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35424 (local address 127.2.56.61:35687)
0928 12:25:39.129882 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:39.129884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:39.130568 (+ 684us) server_negotiation.cc:407] Connection header received
0928 12:25:39.130670 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:39.130671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:39.130682 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:39.130711 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:39.131173 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35424: BlockingRecv error: recv got EOF from 127.0.0.1:35424 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 12:25:39.132126 24577 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:25:39.129591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55780 (local address 127.2.56.62:41815)
0928 12:25:39.129638 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:39.129640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:39.131629 (+ 1989us) server_negotiation.cc:407] Connection header received
0928 12:25:39.131720 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:39.131721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:39.131731 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:39.131758 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:39.132066 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55780: BlockingRecv error: recv got EOF from 127.0.0.1:55780 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 12:25:39.132906 24728 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:25:39.130333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36768 (local address 127.2.56.60:34307)
0928 12:25:39.130374 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:39.130375 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:39.132440 (+ 2065us) server_negotiation.cc:407] Connection header received
0928 12:25:39.132532 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:39.132533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:39.132544 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:39.132571 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:39.132849 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36768: BlockingRecv error: recv got EOF from 127.0.0.1:36768 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 12:25:39.135277 24653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:25:39.134533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35434 (local address 127.2.56.61:35687)
0928 12:25:39.134567 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:39.134569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:39.134764 (+ 195us) server_negotiation.cc:407] Connection header received
0928 12:25:39.134861 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:39.134862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:39.134879 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:39.134906 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:39.135220 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35434: BlockingRecv error: recv got EOF from 127.0.0.1:35434 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 12:25:39.139325 24653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:25:39.138568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35446 (local address 127.2.56.61:35687)
0928 12:25:39.138601 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:39.138603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:39.138858 (+ 255us) server_negotiation.cc:407] Connection header received
0928 12:25:39.138957 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:39.138958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:39.138968 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:39.138995 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:39.139279 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35446: BlockingRecv error: recv got EOF from 127.0.0.1:35446 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 12:25:39.143282 24653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:25:39.142550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35456 (local address 127.2.56.61:35687)
0928 12:25:39.142582 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:39.142584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:39.142812 (+ 228us) server_negotiation.cc:407] Connection header received
0928 12:25:39.142901 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:39.142902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:39.142912 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:39.142950 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:39.143235 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35456: BlockingRecv error: recv got EOF from 127.0.0.1:35456 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 12:25:39.149075 24653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:25:39.148358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35468 (local address 127.2.56.61:35687)
0928 12:25:39.148389 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:39.148391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:39.148600 (+ 209us) server_negotiation.cc:407] Connection header received
0928 12:25:39.148689 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:39.148690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:39.148700 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:39.148726 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:39.149029 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35468: BlockingRecv error: recv got EOF from 127.0.0.1:35468 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 12:25:39.158988 24653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:25:39.158222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35476 (local address 127.2.56.61:35687)
0928 12:25:39.158255 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:39.158257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:39.158478 (+ 221us) server_negotiation.cc:407] Connection header received
0928 12:25:39.158567 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:39.158568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:39.158578 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:39.158604 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:39.158901 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35476: BlockingRecv error: recv got EOF from 127.0.0.1:35476 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 12:25:39.177134 24653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:25:39.176277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35484 (local address 127.2.56.61:35687)
0928 12:25:39.176316 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:39.176318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:39.176568 (+ 250us) server_negotiation.cc:407] Connection header received
0928 12:25:39.176663 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:39.176664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:39.176675 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:39.176704 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:39.177076 (+ 372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35484: BlockingRecv error: recv got EOF from 127.0.0.1:35484 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 12:25:39.211503 24653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:25:39.210536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35500 (local address 127.2.56.61:35687)
0928 12:25:39.210582 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:39.210584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:39.210888 (+ 304us) server_negotiation.cc:407] Connection header received
0928 12:25:39.210995 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:39.210996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:39.211009 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:39.211051 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:39.211442 (+ 391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35500: BlockingRecv error: recv got EOF from 127.0.0.1:35500 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 12:25:39.277588 24653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:25:39.276896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35508 (local address 127.2.56.61:35687)
0928 12:25:39.276973 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:39.276975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:39.277006 (+ 31us) server_negotiation.cc:407] Connection header received
0928 12:25:39.277113 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:39.277114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:39.277125 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:39.277172 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:39.277509 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35508: BlockingRecv error: recv got EOF from 127.0.0.1:35508 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 12:25:39.407301 24653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:25:39.406663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35510 (local address 127.2.56.61:35687)
0928 12:25:39.406733 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:39.406734 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:39.406778 (+ 44us) server_negotiation.cc:407] Connection header received
0928 12:25:39.406851 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:39.406852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:39.406863 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:39.406910 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:39.407236 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35510: BlockingRecv error: recv got EOF from 127.0.0.1:35510 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260928 12:25:39.665318 24653 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:25:39.664534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35522 (local address 127.2.56.61:35687)
0928 12:25:39.664625 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:39.664627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:39.664665 (+ 38us) server_negotiation.cc:407] Connection header received
0928 12:25:39.664759 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:39.664760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:39.664773 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:39.664842 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:39.665232 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35522: BlockingRecv error: recv got EOF from 127.0.0.1:35522 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 12:25:40.027575 24741 catalog_manager.cc:1259] Loaded cluster ID: f12cc0ff973a41279a8fc26d8414c204
I20260928 12:25:40.027678 24741 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e15d437594de481cae848da2e2e2b820: loading cluster ID for follower catalog manager: success
I20260928 12:25:40.028708 24741 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e15d437594de481cae848da2e2e2b820: acquiring CA information for follower catalog manager: success
I20260928 12:25:40.029076 24741 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e15d437594de481cae848da2e2e2b820: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 12:25:40.116842 24788 catalog_manager.cc:1259] Loaded cluster ID: f12cc0ff973a41279a8fc26d8414c204
I20260928 12:25:40.116926 24788 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5e7db1b16bff417e864c4adfce440d5b: loading cluster ID for follower catalog manager: success
I20260928 12:25:40.117785 24788 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5e7db1b16bff417e864c4adfce440d5b: acquiring CA information for follower catalog manager: success
I20260928 12:25:40.118093 24788 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5e7db1b16bff417e864c4adfce440d5b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 12:25:40.181005 24858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:40.178346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35524 (local address 127.2.56.61:35687)
0928 12:25:40.178504 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:40.178507 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:40.178521 (+ 14us) server_negotiation.cc:407] Connection header received
0928 12:25:40.178557 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:40.178558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:40.178634 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:40.178773 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:25:40.178910 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:40.179472 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:40.180241 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:40.180458 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:40.180517 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:40.180638 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:25:40.180782 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:25:40.180844 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:40.180863 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 12:25:40.180885 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":88,"threads_started":1}
I20260928 12:25:40.185295 24864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:40.182902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36772 (local address 127.2.56.60:34307)
0928 12:25:40.183076 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:40.183079 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:40.183086 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:25:40.183132 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:40.183133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:40.183176 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:40.183246 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:25:40.183351 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:40.183851 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:40.184539 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:40.184747 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:40.184799 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:40.184932 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:25:40.185050 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:25:40.185117 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:40.185153 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 12:25:40.185175 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":68,"threads_started":1}
I20260928 12:25:40.189872 24858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:40.186944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35536 (local address 127.2.56.61:35687)
0928 12:25:40.187097 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:40.187099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:40.187205 (+ 106us) server_negotiation.cc:407] Connection header received
0928 12:25:40.187317 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:40.187319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:40.187332 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:40.187364 (+ 32us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:25:40.187540 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:40.188101 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:40.189019 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:40.189364 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:40.189402 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:40.189592 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:25:40.189626 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:25:40.189653 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:40.189760 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 12:25:40.189785 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260928 12:25:40.190059 24864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:40.187183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36778 (local address 127.2.56.60:34307)
0928 12:25:40.187221 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:40.187222 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:40.187382 (+ 160us) server_negotiation.cc:407] Connection header received
0928 12:25:40.187519 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:40.187521 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:40.187535 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:40.187592 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:25:40.187727 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:40.188448 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:40.189243 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:40.189504 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:40.189569 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:40.189724 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:25:40.189766 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:25:40.189812 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:40.189933 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 12:25:40.189978 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 12:25:40.190598 24870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:40.186980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55790 (local address 127.2.56.62:41815)
0928 12:25:40.187155 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:40.187157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:40.187165 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:25:40.187228 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:40.187229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:40.187294 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:40.187391 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:25:40.187523 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:40.188367 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:40.189472 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:40.189750 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:40.189978 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:40.190071 (+ 93us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:25:40.190191 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:25:40.190288 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:40.190465 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 12:25:40.190515 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":64,"threads_started":1}
I20260928 12:25:40.196012 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 24506
I20260928 12:25:40.201941 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 24581
I20260928 12:25:40.207121 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 24656
2026-09-28T12:25:40Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2724 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:25:40 dist-test-slave-920p krb5kdc[24875](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:25:40 dist-test-slave-920p krb5kdc[24875](info): set up 2 sockets
Sep 28 12:25:40 dist-test-slave-920p krb5kdc[24875](info): commencing operation
krb5kdc: starting...
W20260928 12:25:41.157343 2272 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/krb5kdc/test-user.keytab.
Sep 28 12:25:41 dist-test-slave-920p krb5kdc[24875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:25:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:25:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
Entry for principal kudu/127.2.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab.
I20260928 12:25:41.267529 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:34089
--webserver_interface=127.2.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:43165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:34089,127.2.56.61:44149,127.2.56.60:38015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:25:41.344374 24888 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 12:25:41.344578 24888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:41.344601 24888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:41.344624 24888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:41.345729 24888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:25:41.345767 24888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:41.345785 24888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:41.345800 24888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:25:41.345816 24888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:25:41.347267 24888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:43165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:34089,127.2.56.61:44149,127.2.56.60:38015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.62.keytab
--principal=kudu/127.2.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.62:34089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:41.347487 24888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:41.347674 24888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:41.352221 24896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:41.352229 24893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:41.352351 24888 server_base.cc:1034] running on GCE node
W20260928 12:25:41.352237 24894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:41.352667 24888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:41.352916 24888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:41.354043 24888 hybrid_clock.cc:648] HybridClock initialized: now 1790598341354030 us; error 28 us; skew 500 ppm
Sep 28 12:25:41 dist-test-slave-920p krb5kdc[24875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598341, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:41.356218 24888 init.cc:377] Logged in from keytab as kudu/127.2.56.62@KRBTEST.COM (short username kudu)
I20260928 12:25:41.356618 24888 webserver.cc:460] Webserver started at http://127.2.56.62:41857/ using document root <none> and password file <none>
I20260928 12:25:41.356797 24888 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:41.356833 24888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:41.356917 24888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:41.357872 24888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-0/data/instance:
uuid: "713f2698a564411394b8eaacd670a85b"
format_stamp: "Formatted at 2026-09-28 12:25:41 on dist-test-slave-920p"
server_key: "d8782162bcc19fbfcc99d0b377fd16ee"
server_key_iv: "f648d31690662e5b025ef084fdd894c8"
server_key_version: "encryptionkey@0"
I20260928 12:25:41.358152 24888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-0/wal/instance:
uuid: "713f2698a564411394b8eaacd670a85b"
format_stamp: "Formatted at 2026-09-28 12:25:41 on dist-test-slave-920p"
server_key: "d8782162bcc19fbfcc99d0b377fd16ee"
server_key_iv: "f648d31690662e5b025ef084fdd894c8"
server_key_version: "encryptionkey@0"
I20260928 12:25:41.359230 24888 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:25:41.359798 24903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:41.359944 24888 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:25:41.360011 24888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-0/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-0/wal
uuid: "713f2698a564411394b8eaacd670a85b"
format_stamp: "Formatted at 2026-09-28 12:25:41 on dist-test-slave-920p"
server_key: "d8782162bcc19fbfcc99d0b377fd16ee"
server_key_iv: "f648d31690662e5b025ef084fdd894c8"
server_key_version: "encryptionkey@0"
I20260928 12:25:41.360064 24888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:41.374993 24888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:41.376513 24888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:41.376616 24888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:41.380091 24888 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.62:34089
I20260928 12:25:41.380139 24955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.62:34089 every 8 connection(s)
I20260928 12:25:41.380415 24888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-0/data/info.pb
I20260928 12:25:41.381348 24956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:41.381457 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 24888
I20260928 12:25:41.381553 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-0/wal/instance
I20260928 12:25:41.381727 2272 external_mini_cluster.cc:1428] Setting key f2520b4896ebb595e6b3fa995dd73cc4
I20260928 12:25:41.382831 24956 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34089 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.56.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.61@KRBTEST.COM" created.
Sep 28 12:25:41 dist-test-slave-920p krb5kdc[24875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598341, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:25:41.394348 24960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.383628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49956 (local address 127.2.56.62:34089)
0928 12:25:41.383760 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:41.383764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:41.383793 (+ 29us) server_negotiation.cc:407] Connection header received
0928 12:25:41.384252 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:41.384254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:41.384270 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:41.384340 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:41.384876 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.385733 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.386529 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.386815 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.386963 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:41.391384 (+ 4421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:41.391390 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:41.391391 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:41.391397 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:41.392997 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:41.393334 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:41.393337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:41.393337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:41.393416 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:41.393590 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:41.393591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:41.393591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:41.393793 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:41.393860 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:41.394015 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 12:25:41.394124 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260928 12:25:41.394349 24959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.383449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:34089 (local address 127.0.0.1:49956)
0928 12:25:41.383611 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:41.383644 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:41.384114 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:41.384350 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:41.384352 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:41.384649 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:41.384792 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:41.384797 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.385811 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.385811 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:41.386458 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:41.386460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.386957 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.386960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:41.387043 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:41.387477 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:25:41.387484 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:41.391307 (+ 3823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:41.393063 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:41.393069 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:41.393070 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:41.393245 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:41.393469 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:41.393469 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:41.393470 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:41.393540 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:41.393849 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:41.393853 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:41.393911 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 12:25:41.393986 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 12:25:41.394185 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":67,"thread_start_us":73,"threads_started":1}
I20260928 12:25:41.395090 24956 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.56.61" port: 44149 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:25:41.395375 24959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.395277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.61:44149 (local address 127.0.0.1:52572)
0928 12:25:41.395314 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:41.395354 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.61:44149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 12:25:41.395501 24907 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.56.61:44149: connect: Connection refused (error 111)
I20260928 12:25:41.395862 24959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.395777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.61:44149 (local address 127.0.0.1:52588)
0928 12:25:41.395816 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:41.395839 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.61:44149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 12:25:41.395948 24956 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.56.61:44149: Network error: Client connection negotiation failed: client connection to 127.2.56.61:44149: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.61.keytab.
Entry for principal kudu/127.2.56.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.61.keytab.
I20260928 12:25:41.405443 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.61:44149
--webserver_interface=127.2.56.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.61.keytab
--principal=kudu/127.2.56.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:43165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:34089,127.2.56.61:44149,127.2.56.60:38015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:25:41.445161 24956 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.56.61" port: 44149 } attempt: 1
I20260928 12:25:41.445657 24959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.445531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.61:44149 (local address 127.0.0.1:52596)
0928 12:25:41.445591 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:41.445625 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.61:44149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 12:25:41.445967 24959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.445881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.61:44149 (local address 127.0.0.1:52602)
0928 12:25:41.445916 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:41.445950 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.61:44149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 12:25:41.446070 24956 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.56.61:44149: Network error: Client connection negotiation failed: client connection to 127.2.56.61:44149: connect: Connection refused (error 111)
W20260928 12:25:41.484311 24963 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 12:25:41.484536 24963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:41.484553 24963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:41.484573 24963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:41.485733 24963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:25:41.485777 24963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:41.485790 24963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:41.485800 24963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:25:41.485811 24963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:25:41.487311 24963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:43165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:34089,127.2.56.61:44149,127.2.56.60:38015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.61.keytab
--principal=kudu/127.2.56.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.61:44149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.56.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:41.487506 24963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:41.487671 24963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:41.492498 24968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:41.492498 24971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:41.492520 24969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:41.492672 24963 server_base.cc:1034] running on GCE node
I20260928 12:25:41.492933 24963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:41.493239 24963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:41.494402 24963 hybrid_clock.cc:648] HybridClock initialized: now 1790598341494382 us; error 36 us; skew 500 ppm
Sep 28 12:25:41 dist-test-slave-920p krb5kdc[24875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598341, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:41.496677 24963 init.cc:377] Logged in from keytab as kudu/127.2.56.61@KRBTEST.COM (short username kudu)
I20260928 12:25:41.497115 24963 webserver.cc:460] Webserver started at http://127.2.56.61:35435/ using document root <none> and password file <none>
I20260928 12:25:41.497342 24963 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:41.497392 24963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:41.497511 24963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:41.498492 24963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-1/data/instance:
uuid: "3419fc1d6f8b499c96017e7529f4dbd2"
format_stamp: "Formatted at 2026-09-28 12:25:41 on dist-test-slave-920p"
server_key: "6d3b0184e75c1c3f8eb52301ebfa12ef"
server_key_iv: "f190e7aa2ee80d82d909af340b587bf0"
server_key_version: "encryptionkey@0"
I20260928 12:25:41.498807 24963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-1/wal/instance:
uuid: "3419fc1d6f8b499c96017e7529f4dbd2"
format_stamp: "Formatted at 2026-09-28 12:25:41 on dist-test-slave-920p"
server_key: "6d3b0184e75c1c3f8eb52301ebfa12ef"
server_key_iv: "f190e7aa2ee80d82d909af340b587bf0"
server_key_version: "encryptionkey@0"
I20260928 12:25:41.500021 24963 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:25:41.500718 24978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:41.500885 24963 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:25:41.500972 24963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-1/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-1/wal
uuid: "3419fc1d6f8b499c96017e7529f4dbd2"
format_stamp: "Formatted at 2026-09-28 12:25:41 on dist-test-slave-920p"
server_key: "6d3b0184e75c1c3f8eb52301ebfa12ef"
server_key_iv: "f190e7aa2ee80d82d909af340b587bf0"
server_key_version: "encryptionkey@0"
I20260928 12:25:41.501030 24963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:41.514233 24956 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.56.61" port: 44149 } attempt: 2
I20260928 12:25:41.514688 24959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.514556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.61:44149 (local address 127.0.0.1:52618)
0928 12:25:41.514614 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:41.514654 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.61:44149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 12:25:41.515070 24959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.514950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.61:44149 (local address 127.0.0.1:52632)
0928 12:25:41.515012 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:41.515045 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.61:44149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 12:25:41.515161 24956 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.56.61:44149: Network error: Client connection negotiation failed: client connection to 127.2.56.61:44149: connect: Connection refused (error 111)
I20260928 12:25:41.518548 24963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:41.520714 24963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:41.520833 24963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:41.524523 24963 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.61:44149
I20260928 12:25:41.524571 25030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.61:44149 every 8 connection(s)
I20260928 12:25:41.524861 24963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-1/data/info.pb
I20260928 12:25:41.525849 25031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:41.527367 25031 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34089 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:25:41.529621 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 24963
I20260928 12:25:41.529714 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-1/wal/instance
I20260928 12:25:41.529881 2272 external_mini_cluster.cc:1428] Setting key 47112baecd763615a49f092bc1d038c5
Sep 28 12:25:41 dist-test-slave-920p krb5kdc[24875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598341, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.61@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:25:41.536298 24960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.527973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49972 (local address 127.2.56.62:34089)
0928 12:25:41.528044 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:41.528046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:41.528218 (+ 172us) server_negotiation.cc:407] Connection header received
0928 12:25:41.528666 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:41.528667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:41.528680 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:41.528736 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:41.529301 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.529822 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.531132 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.531429 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.531498 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:41.533678 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:41.533680 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:41.533680 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:41.533686 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:41.534941 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:41.535410 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:41.535411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:41.535411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:41.535467 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:41.535727 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:41.535727 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:41.535728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:41.535825 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:41.535872 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:41.536167 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 12:25:41.536233 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 12:25:41.536530 25033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.527962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:34089 (local address 127.0.0.1:49972)
0928 12:25:41.528108 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:41.528144 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:41.528569 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:41.528790 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:41.528793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:41.529015 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:41.529203 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:41.529206 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.529900 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.529901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:41.531042 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:41.531046 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.531509 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.531510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:41.531610 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:41.532049 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:25:41.532056 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:41.533592 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:41.535024 (+ 1432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:41.535041 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:41.535043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:41.535319 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:41.535535 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:41.535536 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:41.535537 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:41.535665 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:41.535880 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:41.535885 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:41.536048 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 12:25:41.536159 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 12:25:41.536311 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
I20260928 12:25:41.537015 25031 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.56.61" port: 44149 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.56.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.56.60@KRBTEST.COM" created.
Sep 28 12:25:41 dist-test-slave-920p krb5kdc[24875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598341, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.61@KRBTEST.COM for kudu/127.2.56.61@KRBTEST.COM
I20260928 12:25:41.545053 25033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.537242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.61:44149 (local address 127.0.0.1:52638)
0928 12:25:41.537302 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:41.537313 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:41.537354 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:41.537564 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:41.537564 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:41.537663 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:41.537711 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:41.537712 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.538571 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.538572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:41.539066 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:41.539067 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.539571 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.539572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:41.539641 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:41.539945 (+ 304us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:25:41.539949 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:41.541199 (+ 1250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:41.543928 (+ 2729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:41.543929 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:41.543929 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:41.544098 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:41.544316 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:41.544316 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:41.544316 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:41.544360 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:41.544787 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:41.544788 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:41.544829 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 12:25:41.544901 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 12:25:41.544981 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 12:25:41.545176 25035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.537310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52638 (local address 127.2.56.61:44149)
0928 12:25:41.537443 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:41.537447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:41.537466 (+ 19us) server_negotiation.cc:407] Connection header received
0928 12:25:41.537485 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:41.537486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:41.537496 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:41.537560 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:41.537753 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.538501 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.539131 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.539481 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.539546 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:41.541312 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:41.541320 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:41.541321 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:41.541328 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:41.543776 (+ 2448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:41.544185 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:41.544188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:41.544189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:41.544254 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:41.544435 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:41.544436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:41.544437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:41.544667 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:41.544773 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:41.544953 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 12:25:41.545045 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":104,"thread_start_us":69,"threads_started":1}
I20260928 12:25:41.545827 25031 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.56.60" port: 38015 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:25:41.546160 25033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.546051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:38015 (local address 127.0.0.1:55110)
0928 12:25:41.546094 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:41.546140 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.60:38015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 12:25:41.546392 24981 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.56.60:38015: connect: Connection refused (error 111)
I20260928 12:25:41.547158 25033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.547061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:38015 (local address 127.0.0.1:55120)
0928 12:25:41.547113 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:41.547140 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.60:38015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 12:25:41.547279 25031 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.56.60:38015: Network error: Client connection negotiation failed: client connection to 127.2.56.60:38015: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.56.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.60.keytab.
Entry for principal kudu/127.2.56.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.60.keytab.
I20260928 12:25:41.555949 2272 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkXRk4O/build/release/bin/kudu
/tmp/dist-test-taskkXRk4O/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.60:38015
--webserver_interface=127.2.56.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.60.keytab
--principal=kudu/127.2.56.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.56.20:43165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.56.62:34089,127.2.56.61:44149,127.2.56.60:38015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:25:41.596534 25031 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.56.60" port: 38015 } attempt: 1
I20260928 12:25:41.597051 25033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.596923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:38015 (local address 127.0.0.1:55130)
0928 12:25:41.596987 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:41.597019 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.60:38015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 12:25:41.597419 25033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.597347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:38015 (local address 127.0.0.1:55134)
0928 12:25:41.597379 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:41.597402 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.60:38015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 12:25:41.597501 25031 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.56.60:38015: Network error: Client connection negotiation failed: client connection to 127.2.56.60:38015: connect: Connection refused (error 111)
I20260928 12:25:41.606264 24956 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.56.61" port: 44149 } attempt: 3
Sep 28 12:25:41 dist-test-slave-920p krb5kdc[24875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598341, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for kudu/127.2.56.61@KRBTEST.COM
I20260928 12:25:41.612818 25035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.606688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52652 (local address 127.2.56.61:44149)
0928 12:25:41.606733 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:41.606735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:41.606745 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:25:41.606792 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:41.606793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:41.606802 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:41.606850 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:41.607159 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.607661 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.608356 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.608572 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.608627 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:41.610587 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:41.610588 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:41.610589 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:41.610595 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:41.611833 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:41.612136 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:41.612137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:41.612138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:41.612189 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:41.612343 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:41.612344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:41.612344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:41.612452 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:41.612506 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:41.612643 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 12:25:41.612718 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:25:41.612818 24959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.606631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.61:44149 (local address 127.0.0.1:52652)
0928 12:25:41.606677 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:41.606688 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:41.606735 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:41.606860 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:41.606861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:41.607018 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:41.607089 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:41.607091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.607744 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.607745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:41.608284 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:41.608286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.608631 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.608633 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:41.608707 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:41.609110 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:25:41.609115 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:41.610502 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:41.611910 (+ 1408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:41.611911 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:41.611912 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:41.612076 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:41.612240 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:41.612240 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:41.612241 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:41.612296 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:41.612502 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:41.612503 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:41.612559 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 12:25:41.612616 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 12:25:41.612718 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 12:25:41.613286 24956 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.56.60" port: 38015 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:25:41.613616 24959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.613519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:38015 (local address 127.0.0.1:55142)
0928 12:25:41.613561 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:41.613588 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.60:38015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 12:25:41.613873 24959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.613794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:38015 (local address 127.0.0.1:55154)
0928 12:25:41.613829 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:41.613851 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.60:38015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 12:25:41.613984 24956 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.56.60:38015: Network error: Client connection negotiation failed: client connection to 127.2.56.60:38015: connect: Connection refused (error 111)
W20260928 12:25:41.636322 25038 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 12:25:41.636512 25038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:41.636528 25038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:41.636546 25038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:41.637631 25038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:25:41.637668 25038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:41.637681 25038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:41.637692 25038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:25:41.637702 25038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:25:41.639123 25038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.56.20:43165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.56.62:34089,127.2.56.61:44149,127.2.56.60:38015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/krb5kdc/kudu_127.2.56.60.keytab
--principal=kudu/127.2.56.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.56.60:38015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.56.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:41.639313 25038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:41.639492 25038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:41.644253 25046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:41.644263 25044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:41.644522 25038 server_base.cc:1034] running on GCE node
W20260928 12:25:41.644271 25043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:41.644785 25038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:41.645087 25038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:41.645130 24956 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.56.60" port: 38015 } attempt: 1
I20260928 12:25:41.645512 24959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.645385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:38015 (local address 127.0.0.1:55166)
0928 12:25:41.645450 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:41.645481 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.60:38015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 12:25:41.645793 24959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.645741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:38015 (local address 127.0.0.1:55172)
0928 12:25:41.645761 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:41.645779 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.60:38015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260928 12:25:41.645859 24956 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.56.60:38015: Network error: Client connection negotiation failed: client connection to 127.2.56.60:38015: connect: Connection refused (error 111)
I20260928 12:25:41.646235 25038 hybrid_clock.cc:648] HybridClock initialized: now 1790598341646202 us; error 55 us; skew 500 ppm
Sep 28 12:25:41 dist-test-slave-920p krb5kdc[24875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598341, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:41.648638 25038 init.cc:377] Logged in from keytab as kudu/127.2.56.60@KRBTEST.COM (short username kudu)
I20260928 12:25:41.649072 25038 webserver.cc:460] Webserver started at http://127.2.56.60:43793/ using document root <none> and password file <none>
I20260928 12:25:41.649310 25038 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:41.649358 25038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:41.649451 25038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:41.650455 25038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-2/data/instance:
uuid: "34c516a6495c436a9ec4dd2083f14e62"
format_stamp: "Formatted at 2026-09-28 12:25:41 on dist-test-slave-920p"
server_key: "87ceec08baf769503b0d6a4e3cda3249"
server_key_iv: "b902b678d754b79d24fd66ae9a60080b"
server_key_version: "encryptionkey@0"
I20260928 12:25:41.650764 25038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-2/wal/instance:
uuid: "34c516a6495c436a9ec4dd2083f14e62"
format_stamp: "Formatted at 2026-09-28 12:25:41 on dist-test-slave-920p"
server_key: "87ceec08baf769503b0d6a4e3cda3249"
server_key_iv: "b902b678d754b79d24fd66ae9a60080b"
server_key_version: "encryptionkey@0"
I20260928 12:25:41.651911 25038 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 12:25:41.652563 25053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:41.652758 25038 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:25:41.652840 25038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-2/data,/tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-2/wal
uuid: "34c516a6495c436a9ec4dd2083f14e62"
format_stamp: "Formatted at 2026-09-28 12:25:41 on dist-test-slave-920p"
server_key: "87ceec08baf769503b0d6a4e3cda3249"
server_key_iv: "b902b678d754b79d24fd66ae9a60080b"
server_key_version: "encryptionkey@0"
I20260928 12:25:41.652897 25038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:41.665652 25031 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.56.60" port: 38015 } attempt: 2
I20260928 12:25:41.666069 25033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.665952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:38015 (local address 127.0.0.1:55184)
0928 12:25:41.666005 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:41.666042 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.60:38015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 12:25:41.666368 25033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.666291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:38015 (local address 127.0.0.1:55196)
0928 12:25:41.666328 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:41.666352 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.56.60:38015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 12:25:41.666446 25031 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.56.60:38015: Network error: Client connection negotiation failed: client connection to 127.2.56.60:38015: connect: Connection refused (error 111)
I20260928 12:25:41.669997 25038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:41.671923 25038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:41.672030 25038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:41.676366 25038 rpc_server.cc:307] RPC server started. Bound to: 127.2.56.60:38015
I20260928 12:25:41.676396 25105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.56.60:38015 every 8 connection(s)
I20260928 12:25:41.676717 25038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-2/data/info.pb
I20260928 12:25:41.677695 25106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:41.679154 25106 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34089 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:25:41.680395 2272 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkXRk4O/build/release/bin/kudu as pid 25038
I20260928 12:25:41.680502 2272 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkXRk4O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598072638673-2272-0/minicluster-data/master-2/wal/instance
I20260928 12:25:41.680691 2272 external_mini_cluster.cc:1428] Setting key ade4c62290dd437a1127406416f01863
Sep 28 12:25:41 dist-test-slave-920p krb5kdc[24875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598341, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.60@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
Sep 28 12:25:41 dist-test-slave-920p krb5kdc[24875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.60@KRBTEST.COM
I20260928 12:25:41.687608 24960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.679781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49982 (local address 127.2.56.62:34089)
0928 12:25:41.679856 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:41.679858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:41.680005 (+ 147us) server_negotiation.cc:407] Connection header received
0928 12:25:41.680470 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:41.680472 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:41.680485 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:41.680552 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:41.681252 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.681752 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.682697 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.682890 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.682953 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:41.685022 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:41.685023 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:41.685023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:41.685030 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:41.686357 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:41.686773 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:41.686774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:41.686774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:41.686824 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:41.687032 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:41.687033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:41.687033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:41.687144 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:41.687198 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:41.687446 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 12:25:41.687519 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 12:25:41.687759 25108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.679736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.62:34089 (local address 127.0.0.1:49982)
0928 12:25:41.679878 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:41.679912 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:41.680365 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:41.680615 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:41.680618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:41.680892 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:41.681168 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:41.681174 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.681830 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.681832 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:41.682620 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:41.682622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.682976 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.682978 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:41.683096 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:41.683520 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:25:41.683528 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:41.684943 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:41.686424 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:41.686431 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:41.686433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:41.686711 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:41.686876 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:41.686876 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:41.686877 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:41.686976 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:41.687197 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:41.687204 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:41.687333 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 12:25:41.687418 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 12:25:41.687560 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
I20260928 12:25:41.688377 25106 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.56.61" port: 44149 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:25:41.688467 25110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.681065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55212 (local address 127.2.56.60:38015)
0928 12:25:41.681227 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:41.681231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:41.681250 (+ 19us) server_negotiation.cc:407] Connection header received
0928 12:25:41.681325 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:41.681328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:41.681350 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:41.681414 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:41.681948 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.682719 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.683399 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.683688 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.683752 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:41.686052 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:41.686058 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:41.686058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:41.686064 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:41.687435 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:41.687771 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:41.687774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:41.687774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:41.687826 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:41.688008 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:41.688009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:41.688009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:41.688149 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:41.688264 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:41.688330 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 12:25:41.688397 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":90,"threads_started":1}
Sep 28 12:25:41 dist-test-slave-920p krb5kdc[24875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598341, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.60@KRBTEST.COM for kudu/127.2.56.61@KRBTEST.COM
Sep 28 12:25:41 dist-test-slave-920p krb5kdc[24875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.62@KRBTEST.COM
I20260928 12:25:41.695323 25108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.688600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.61:44149 (local address 127.0.0.1:52654)
0928 12:25:41.688689 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:41.688698 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:41.688917 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:41.689104 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:41.689105 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:41.689235 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:41.689302 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:41.689304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.689856 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.689857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:41.690589 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:41.690591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.691094 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.691095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:41.691154 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:41.691507 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:25:41.691511 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:41.693051 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:41.694458 (+ 1407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:41.694459 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:41.694459 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:41.694609 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:41.694769 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:41.694770 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:41.694770 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:41.694821 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:41.695023 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:41.695024 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:41.695088 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 12:25:41.695143 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 12:25:41.695229 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 12:25:41.695326 25035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.688705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52654 (local address 127.2.56.61:44149)
0928 12:25:41.688754 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:41.688756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:41.688766 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:25:41.689020 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:41.689022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:41.689033 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:41.689099 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:41.689363 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.689805 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.690674 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.690999 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.691076 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:41.693137 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:41.693139 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:41.693139 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:41.693147 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:41.694392 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:41.694668 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:41.694669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:41.694670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:41.694718 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:41.694871 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:41.694871 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:41.694872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:41.694973 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:41.695024 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:41.695174 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 12:25:41.695240 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:25:41.695676 25106 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.56.60" port: 38015 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:25:41.696484 24960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.689255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49992 (local address 127.2.56.62:34089)
0928 12:25:41.689314 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:41.689316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:41.689322 (+ 6us) server_negotiation.cc:407] Connection header received
0928 12:25:41.689339 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:41.689340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:41.689348 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:41.689398 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:41.689788 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.690483 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.691404 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.691696 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.691768 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:41.694144 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:41.694145 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:41.694145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:41.694152 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:41.695330 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:41.695845 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:41.695846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:41.695846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:41.695896 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:41.696061 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:41.696062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:41.696062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:41.696157 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:41.696308 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:41.696342 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 12:25:41.696404 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 12:25:41 dist-test-slave-920p krb5kdc[24875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598341, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.60@KRBTEST.COM for kudu/127.2.56.60@KRBTEST.COM
Sep 28 12:25:41 dist-test-slave-920p krb5kdc[24875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.56.61@KRBTEST.COM
I20260928 12:25:41.701576 25108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.695894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:38015 (local address 127.0.0.1:55226)
0928 12:25:41.695949 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:41.695957 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:41.695994 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:41.696093 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:41.696094 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:41.696208 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:41.696231 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:41.696232 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.696964 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.696965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:41.697568 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:41.697570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.697891 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.697892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:41.697943 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:41.698269 (+ 326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:25:41.698273 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:41.699338 (+ 1065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:41.700652 (+ 1314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:41.700652 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:41.700653 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:41.700792 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:41.700970 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:41.700971 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:41.700971 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:41.701022 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:41.701259 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:41.701260 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:41.701322 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 12:25:41.701387 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 12:25:41.701471 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 12:25:41.701613 25110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.695922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55226 (local address 127.2.56.60:38015)
0928 12:25:41.695952 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:41.695954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:41.695983 (+ 29us) server_negotiation.cc:407] Connection header received
0928 12:25:41.696040 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:41.696041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:41.696050 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:41.696092 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:41.696286 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.696896 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.697644 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.697837 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.697881 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:41.699395 (+ 1514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:41.699396 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:41.699396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:41.699403 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:41.700583 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:41.700855 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:41.700856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:41.700856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:41.700906 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:41.701089 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:41.701090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:41.701090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:41.701200 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:41.701282 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:41.701419 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 12:25:41.701512 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":13}
I20260928 12:25:41.702849 25106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34c516a6495c436a9ec4dd2083f14e62: Bootstrap starting.
I20260928 12:25:41.703646 25106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34c516a6495c436a9ec4dd2083f14e62: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:41.703894 25106 log.cc:824] T 00000000000000000000000000000000 P 34c516a6495c436a9ec4dd2083f14e62: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:41.704627 25106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34c516a6495c436a9ec4dd2083f14e62: No bootstrap required, opened a new log
I20260928 12:25:41.705891 25035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.697037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52656 (local address 127.2.56.61:44149)
0928 12:25:41.697069 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:41.697071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:41.697077 (+ 6us) server_negotiation.cc:407] Connection header received
0928 12:25:41.697110 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:41.697111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:41.697119 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:41.697169 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:41.697602 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.698139 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.698830 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.699005 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.699059 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:41.700939 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:41.700940 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:41.700940 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:41.700946 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:41.704462 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:41.704819 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:41.704820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:41.704820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:41.704867 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:41.705073 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:41.705074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:41.705074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:41.705170 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:41.705673 (+ 503us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:41.705715 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 12:25:41.705786 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 12:25:41.706646 25106 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34c516a6495c436a9ec4dd2083f14e62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "713f2698a564411394b8eaacd670a85b" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34089 } } peers { permanent_uuid: "3419fc1d6f8b499c96017e7529f4dbd2" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 44149 } } peers { permanent_uuid: "34c516a6495c436a9ec4dd2083f14e62" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 38015 } }
I20260928 12:25:41.706799 25106 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34c516a6495c436a9ec4dd2083f14e62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34c516a6495c436a9ec4dd2083f14e62, State: Initialized, Role: FOLLOWER
I20260928 12:25:41.706882 25106 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34c516a6495c436a9ec4dd2083f14e62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "713f2698a564411394b8eaacd670a85b" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34089 } } peers { permanent_uuid: "3419fc1d6f8b499c96017e7529f4dbd2" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 44149 } } peers { permanent_uuid: "34c516a6495c436a9ec4dd2083f14e62" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 38015 } }
I20260928 12:25:41.707131 25112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34c516a6495c436a9ec4dd2083f14e62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "713f2698a564411394b8eaacd670a85b" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34089 } } peers { permanent_uuid: "3419fc1d6f8b499c96017e7529f4dbd2" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 44149 } } peers { permanent_uuid: "34c516a6495c436a9ec4dd2083f14e62" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 38015 } } }
I20260928 12:25:41.707196 25106 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34c516a6495c436a9ec4dd2083f14e62 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:25:41.707221 25112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34c516a6495c436a9ec4dd2083f14e62 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:25:41.707299 2272 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 12:25:41.708954 25123 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 34c516a6495c436a9ec4dd2083f14e62: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:25:41.709012 25123 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:25:41.708487 24930 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "713f2698a564411394b8eaacd670a85b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49992
I20260928 12:25:41.709154 25038 master_runner.cc:186] Error getting master registration for 127.2.56.62:34089: OK, instance_id { permanent_uuid: "713f2698a564411394b8eaacd670a85b" instance_seqno: 1790598341379164 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 12:25:41.711992 25038 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 12:25:41.721114 24956 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.56.60" port: 38015 } attempt: 2
Sep 28 12:25:41 dist-test-slave-920p krb5kdc[24875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598341, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.62@KRBTEST.COM for kudu/127.2.56.60@KRBTEST.COM
I20260928 12:25:41.729831 24959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.721426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:38015 (local address 127.0.0.1:55236)
0928 12:25:41.721479 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:41.721489 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:41.721528 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:41.721649 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:41.721650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:41.721820 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:41.721869 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:41.721871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.722757 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.722758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:41.723509 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:41.723512 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.724324 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.724325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:41.724397 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:41.724849 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:25:41.724854 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:41.726619 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:41.728538 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:41.728539 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:41.728539 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:41.728759 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:41.728957 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:41.728958 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:41.728959 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:41.729016 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:41.729519 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:41.729520 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:41.729572 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 12:25:41.729634 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 12:25:41.729733 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 12:25:41.730171 25110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.721499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55236 (local address 127.2.56.60:38015)
0928 12:25:41.721550 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:41.721552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:41.721561 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:25:41.721583 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:41.721583 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:41.721592 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:41.721640 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:41.721982 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.722683 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.723910 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.724189 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.724257 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:41.726723 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:41.726725 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:41.726725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:41.726732 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:41.728441 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:41.728837 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:41.728838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:41.728839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:41.728894 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:41.729173 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:41.729174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:41.729175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:41.729400 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:41.729487 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:41.730006 (+ 519us) server_negotiation.cc:299] Negotiation successful
0928 12:25:41.730087 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 12:25:41.731751 24956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 713f2698a564411394b8eaacd670a85b: Bootstrap starting.
I20260928 12:25:41.732507 24956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 713f2698a564411394b8eaacd670a85b: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:41.732821 24956 log.cc:824] T 00000000000000000000000000000000 P 713f2698a564411394b8eaacd670a85b: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:41.733628 24956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 713f2698a564411394b8eaacd670a85b: No bootstrap required, opened a new log
I20260928 12:25:41.735149 24956 raft_consensus.cc:348] T 00000000000000000000000000000000 P 713f2698a564411394b8eaacd670a85b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "713f2698a564411394b8eaacd670a85b" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34089 } } peers { permanent_uuid: "3419fc1d6f8b499c96017e7529f4dbd2" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 44149 } } peers { permanent_uuid: "34c516a6495c436a9ec4dd2083f14e62" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 38015 } }
I20260928 12:25:41.735448 24956 raft_consensus.cc:729] T 00000000000000000000000000000000 P 713f2698a564411394b8eaacd670a85b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 713f2698a564411394b8eaacd670a85b, State: Initialized, Role: FOLLOWER
I20260928 12:25:41.735535 24956 consensus_queue.cc:260] T 00000000000000000000000000000000 P 713f2698a564411394b8eaacd670a85b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "713f2698a564411394b8eaacd670a85b" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34089 } } peers { permanent_uuid: "3419fc1d6f8b499c96017e7529f4dbd2" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 44149 } } peers { permanent_uuid: "34c516a6495c436a9ec4dd2083f14e62" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 38015 } }
I20260928 12:25:41.735848 24956 sys_catalog.cc:564] T 00000000000000000000000000000000 P 713f2698a564411394b8eaacd670a85b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:25:41.737372 24930 raft_consensus.cc:482] T 00000000000000000000000000000000 P 713f2698a564411394b8eaacd670a85b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 12:25:41.737442 24930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 713f2698a564411394b8eaacd670a85b [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:41.738269 24930 raft_consensus.cc:504] T 00000000000000000000000000000000 P 713f2698a564411394b8eaacd670a85b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "713f2698a564411394b8eaacd670a85b" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34089 } } peers { permanent_uuid: "3419fc1d6f8b499c96017e7529f4dbd2" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 44149 } } peers { permanent_uuid: "34c516a6495c436a9ec4dd2083f14e62" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 38015 } }
I20260928 12:25:41.738528 24930 leader_election.cc:288] T 00000000000000000000000000000000 P 713f2698a564411394b8eaacd670a85b [CANDIDATE]: Term 1 election: Requested vote from peers 3419fc1d6f8b499c96017e7529f4dbd2 (127.2.56.61:44149), 34c516a6495c436a9ec4dd2083f14e62 (127.2.56.60:38015)
I20260928 12:25:41.739076 25080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "713f2698a564411394b8eaacd670a85b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "34c516a6495c436a9ec4dd2083f14e62"
I20260928 12:25:41.739216 25080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34c516a6495c436a9ec4dd2083f14e62 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:41.740147 25080 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 34c516a6495c436a9ec4dd2083f14e62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 713f2698a564411394b8eaacd670a85b in term 1.
I20260928 12:25:41.740471 24906 leader_election.cc:302] T 00000000000000000000000000000000 P 713f2698a564411394b8eaacd670a85b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34c516a6495c436a9ec4dd2083f14e62, 713f2698a564411394b8eaacd670a85b; no voters:
I20260928 12:25:41.740739 25144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 713f2698a564411394b8eaacd670a85b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:25:41.740958 25144 raft_consensus.cc:686] T 00000000000000000000000000000000 P 713f2698a564411394b8eaacd670a85b [term 1 LEADER]: Becoming Leader. State: Replica: 713f2698a564411394b8eaacd670a85b, State: Running, Role: LEADER
I20260928 12:25:41.741096 25144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 713f2698a564411394b8eaacd670a85b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "713f2698a564411394b8eaacd670a85b" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34089 } } peers { permanent_uuid: "3419fc1d6f8b499c96017e7529f4dbd2" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 44149 } } peers { permanent_uuid: "34c516a6495c436a9ec4dd2083f14e62" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 38015 } }
I20260928 12:25:41.741758 25144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 713f2698a564411394b8eaacd670a85b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 713f2698a564411394b8eaacd670a85b. Latest consensus state: current_term: 1 leader_uuid: "713f2698a564411394b8eaacd670a85b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "713f2698a564411394b8eaacd670a85b" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34089 } } peers { permanent_uuid: "3419fc1d6f8b499c96017e7529f4dbd2" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 44149 } } peers { permanent_uuid: "34c516a6495c436a9ec4dd2083f14e62" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 38015 } } }
I20260928 12:25:41.741894 25144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 713f2698a564411394b8eaacd670a85b [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:41.742214 25147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:25:41.742754 25147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:25:41.740602 24888 master_runner.cc:186] Error getting master registration for 127.2.56.61:44149: OK, instance_id { permanent_uuid: "3419fc1d6f8b499c96017e7529f4dbd2" instance_seqno: 1790598341523539 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 12:25:41.743423 24888 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 12:25:41.743713 25126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 713f2698a564411394b8eaacd670a85b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "713f2698a564411394b8eaacd670a85b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "713f2698a564411394b8eaacd670a85b" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34089 } } peers { permanent_uuid: "3419fc1d6f8b499c96017e7529f4dbd2" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 44149 } } peers { permanent_uuid: "34c516a6495c436a9ec4dd2083f14e62" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 38015 } } }
I20260928 12:25:41.743800 25126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 713f2698a564411394b8eaacd670a85b [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:41.744874 25080 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 34c516a6495c436a9ec4dd2083f14e62 [term 1 FOLLOWER]: Refusing update from remote peer 713f2698a564411394b8eaacd670a85b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:25:41.745223 25126 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 713f2698a564411394b8eaacd670a85b [LEADER]: Connected to new peer: Peer: permanent_uuid: "34c516a6495c436a9ec4dd2083f14e62" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 38015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:25:41.747293 25112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34c516a6495c436a9ec4dd2083f14e62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 713f2698a564411394b8eaacd670a85b. Latest consensus state: current_term: 1 leader_uuid: "713f2698a564411394b8eaacd670a85b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "713f2698a564411394b8eaacd670a85b" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34089 } } peers { permanent_uuid: "3419fc1d6f8b499c96017e7529f4dbd2" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 44149 } } peers { permanent_uuid: "34c516a6495c436a9ec4dd2083f14e62" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 38015 } } }
I20260928 12:25:41.747391 25112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34c516a6495c436a9ec4dd2083f14e62 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:25:41.751029 25110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.740133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55242 (local address 127.2.56.60:38015)
0928 12:25:41.740174 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:41.740176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:41.740422 (+ 246us) server_negotiation.cc:407] Connection header received
0928 12:25:41.740584 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:41.740585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:41.740611 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:41.740640 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:41.742089 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.742759 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.744306 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.744602 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.744642 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:41.747752 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:41.747753 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:41.747754 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:41.747760 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:41.749319 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:41.749986 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:41.749987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:41.749987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:41.750042 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:41.750332 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:41.750333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:41.750333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:41.750484 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:41.750556 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:41.750868 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 12:25:41.750943 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:25:41.752446 25145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 713f2698a564411394b8eaacd670a85b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "713f2698a564411394b8eaacd670a85b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "713f2698a564411394b8eaacd670a85b" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34089 } } peers { permanent_uuid: "3419fc1d6f8b499c96017e7529f4dbd2" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 44149 } } peers { permanent_uuid: "34c516a6495c436a9ec4dd2083f14e62" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 38015 } } }
I20260928 12:25:41.752532 25145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 713f2698a564411394b8eaacd670a85b [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:41.753130 25112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34c516a6495c436a9ec4dd2083f14e62 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "713f2698a564411394b8eaacd670a85b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "713f2698a564411394b8eaacd670a85b" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34089 } } peers { permanent_uuid: "3419fc1d6f8b499c96017e7529f4dbd2" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 44149 } } peers { permanent_uuid: "34c516a6495c436a9ec4dd2083f14e62" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 38015 } } }
I20260928 12:25:41.753217 25112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34c516a6495c436a9ec4dd2083f14e62 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:25:41.753479 25151 mvcc.cc:205] Tried to move back new op lower bound from 7334290807784382464 to 7334290807772684288. Current Snapshot: MvccSnapshot[applied={T|T < 7334290807784382464}]
I20260928 12:25:41.757558 25031 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.56.60" port: 38015 } attempt: 3
I20260928 12:25:41.757391 25149 mvcc.cc:205] Tried to move back new op lower bound from 7334290807784382464 to 7334290807772684288. Current Snapshot: MvccSnapshot[applied={T|T < 7334290807784382464}]
I20260928 12:25:41.760071 24960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.745470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50002 (local address 127.2.56.62:34089)
0928 12:25:41.745507 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:41.745509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:41.751526 (+ 6017us) server_negotiation.cc:407] Connection header received
0928 12:25:41.751638 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:41.751639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:41.751650 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:41.751718 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:41.752860 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.753604 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.754571 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.754893 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.754969 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:41.756365 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:41.756366 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:41.756366 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:41.756373 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:41.758065 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:41.758987 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:41.758988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:41.758989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:41.759064 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:41.759349 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:41.759350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:41.759350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:41.759493 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:41.759562 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:41.759917 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 12:25:41.759991 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:25:41.760511 25147 catalog_manager.cc:1347] Generated new cluster ID: 6fbdf63fb42e4044a360d97916680aff
I20260928 12:25:41.760553 25147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:25:41.768252 25147 catalog_manager.cc:1370] Generated new certificate authority record
Sep 28 12:25:41 dist-test-slave-920p krb5kdc[24875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598341, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.56.61@KRBTEST.COM for kudu/127.2.56.60@KRBTEST.COM
I20260928 12:25:41.769531 25147 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:25:41.776474 25147 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 713f2698a564411394b8eaacd670a85b: Generated new TSK 0
I20260928 12:25:41.776631 25147 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:25:41.782366 25033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.757804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.56.60:38015 (local address 127.0.0.1:55256)
0928 12:25:41.757911 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:41.757920 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:41.757956 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:41.758179 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:41.758180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:41.758337 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:41.758396 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:41.758398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.759361 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.759362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:41.760320 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:41.760323 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.760803 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.760804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:41.760866 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:41.767724 (+ 6858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:25:41.767729 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:41.769142 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:41.775554 (+ 6412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:41.775554 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:41.775555 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:41.775745 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:41.776443 (+ 698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:41.776444 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:41.776444 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:41.781513 (+ 5069us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:41.782030 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:41.782031 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:41.782074 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 12:25:41.782129 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 12:25:41.782263 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260928 12:25:41.776283 25035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.740010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52658 (local address 127.2.56.61:44149)
0928 12:25:41.740050 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:41.740052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:41.745615 (+ 5563us) server_negotiation.cc:407] Connection header received
0928 12:25:41.745712 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:41.745713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:41.745724 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:41.745751 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:41.746456 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.749161 (+ 2705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.750115 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.750423 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.750492 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:41.768928 (+ 18436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:41.768941 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:41.768942 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:41.768949 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:41.774550 (+ 5601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:41.774918 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:41.774920 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:41.774920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:41.774992 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:41.775303 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:41.775303 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:41.775304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:41.775428 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:41.775530 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:41.776108 (+ 578us) server_negotiation.cc:299] Negotiation successful
0928 12:25:41.776186 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:25:41.783038 25110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:41.758036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55256 (local address 127.2.56.60:38015)
0928 12:25:41.758070 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:41.758071 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:41.758077 (+ 6us) server_negotiation.cc:407] Connection header received
0928 12:25:41.758094 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:41.758095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:41.758103 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:41.758144 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:41.758478 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.759273 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.760417 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:41.760690 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:41.760745 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:41.769233 (+ 8488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:41.769234 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:41.769235 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:41.769241 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:41.775496 (+ 6255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:41.776311 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:41.776313 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:41.776313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:41.776371 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:41.781622 (+ 5251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:41.781623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:41.781623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:41.781871 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:41.782577 (+ 706us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:41.782635 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 12:25:41.782706 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:25:41.784572 25031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3419fc1d6f8b499c96017e7529f4dbd2: Bootstrap starting.
I20260928 12:25:41.785254 25031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3419fc1d6f8b499c96017e7529f4dbd2: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:41.785598 25031 log.cc:824] T 00000000000000000000000000000000 P 3419fc1d6f8b499c96017e7529f4dbd2: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:41.787180 25031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3419fc1d6f8b499c96017e7529f4dbd2: No bootstrap required, opened a new log
I20260928 12:25:41.789173 25031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3419fc1d6f8b499c96017e7529f4dbd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "713f2698a564411394b8eaacd670a85b" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34089 } } peers { permanent_uuid: "3419fc1d6f8b499c96017e7529f4dbd2" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 44149 } } peers { permanent_uuid: "34c516a6495c436a9ec4dd2083f14e62" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 38015 } }
I20260928 12:25:41.789335 25031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3419fc1d6f8b499c96017e7529f4dbd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3419fc1d6f8b499c96017e7529f4dbd2, State: Initialized, Role: FOLLOWER
I20260928 12:25:41.789413 25031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3419fc1d6f8b499c96017e7529f4dbd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "713f2698a564411394b8eaacd670a85b" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34089 } } peers { permanent_uuid: "3419fc1d6f8b499c96017e7529f4dbd2" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 44149 } } peers { permanent_uuid: "34c516a6495c436a9ec4dd2083f14e62" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 38015 } }
I20260928 12:25:41.789707 25031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3419fc1d6f8b499c96017e7529f4dbd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:25:41.789785 25004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3419fc1d6f8b499c96017e7529f4dbd2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:41.791093 25004 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3419fc1d6f8b499c96017e7529f4dbd2 [term 1 FOLLOWER]: Refusing update from remote peer 713f2698a564411394b8eaacd670a85b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:25:41.791303 25005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "713f2698a564411394b8eaacd670a85b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3419fc1d6f8b499c96017e7529f4dbd2"
I20260928 12:25:41.792014 25005 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3419fc1d6f8b499c96017e7529f4dbd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 713f2698a564411394b8eaacd670a85b in term 1.
I20260928 12:25:41.792332 25144 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 713f2698a564411394b8eaacd670a85b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3419fc1d6f8b499c96017e7529f4dbd2" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 44149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 12:25:41.795497 25171 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3419fc1d6f8b499c96017e7529f4dbd2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:25:41.795545 25171 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:25:41.797067 25156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3419fc1d6f8b499c96017e7529f4dbd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "713f2698a564411394b8eaacd670a85b" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34089 } } peers { permanent_uuid: "3419fc1d6f8b499c96017e7529f4dbd2" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 44149 } } peers { permanent_uuid: "34c516a6495c436a9ec4dd2083f14e62" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 38015 } } }
I20260928 12:25:41.797341 25156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3419fc1d6f8b499c96017e7529f4dbd2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:25:41.797438 25156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3419fc1d6f8b499c96017e7529f4dbd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 713f2698a564411394b8eaacd670a85b. Latest consensus state: current_term: 1 leader_uuid: "713f2698a564411394b8eaacd670a85b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "713f2698a564411394b8eaacd670a85b" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34089 } } peers { permanent_uuid: "3419fc1d6f8b499c96017e7529f4dbd2" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 44149 } } peers { permanent_uuid: "34c516a6495c436a9ec4dd2083f14e62" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 38015 } } }
I20260928 12:25:41.797534 25156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3419fc1d6f8b499c96017e7529f4dbd2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:25:41.797610 25156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3419fc1d6f8b499c96017e7529f4dbd2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "713f2698a564411394b8eaacd670a85b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "713f2698a564411394b8eaacd670a85b" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34089 } } peers { permanent_uuid: "3419fc1d6f8b499c96017e7529f4dbd2" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 44149 } } peers { permanent_uuid: "34c516a6495c436a9ec4dd2083f14e62" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 38015 } } }
I20260928 12:25:41.797693 25156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3419fc1d6f8b499c96017e7529f4dbd2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:25:41.805855 25158 mvcc.cc:205] Tried to move back new op lower bound from 7334290807908093952 to 7334290807772684288. Current Snapshot: MvccSnapshot[applied={T|T < 7334290807784382464}]
W20260928 12:25:41.807358 25035 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:25:41.806284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52666 (local address 127.2.56.61:44149)
0928 12:25:41.806319 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:41.806322 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:41.806756 (+ 434us) server_negotiation.cc:407] Connection header received
0928 12:25:41.806886 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:41.806887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:41.806897 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:41.806952 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:41.807307 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52666: BlockingRecv error: recv got EOF from 127.0.0.1:52666 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 12:25:41.807948 25110 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:25:41.806821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55266 (local address 127.2.56.60:38015)
0928 12:25:41.806854 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:41.806856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:41.807535 (+ 679us) server_negotiation.cc:407] Connection header received
0928 12:25:41.807580 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:41.807581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:41.807590 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:41.807655 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:41.807892 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55266: BlockingRecv error: recv got EOF from 127.0.0.1:55266 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 12:25:41.808807 24960 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:25:41.807250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50018 (local address 127.2.56.62:34089)
0928 12:25:41.807305 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:41.807307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:41.808207 (+ 900us) server_negotiation.cc:407] Connection header received
0928 12:25:41.808291 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:41.808292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:41.808302 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:41.808344 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:41.808752 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50018: BlockingRecv error: recv got EOF from 127.0.0.1:50018 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260928 12:25:41.810781 25035 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:25:41.809835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52668 (local address 127.2.56.61:44149)
0928 12:25:41.809871 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:41.809872 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:41.810314 (+ 442us) server_negotiation.cc:407] Connection header received
0928 12:25:41.810420 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:41.810421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:41.810435 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:41.810462 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:41.810740 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52668: BlockingRecv error: recv got EOF from 127.0.0.1:52668 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 12:25:41.813623 25035 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:25:41.813013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52680 (local address 127.2.56.61:44149)
0928 12:25:41.813043 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:41.813045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:41.813142 (+ 97us) server_negotiation.cc:407] Connection header received
0928 12:25:41.813224 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:41.813225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:41.813244 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:41.813293 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:41.813570 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52680: BlockingRecv error: recv got EOF from 127.0.0.1:52680 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260928 12:25:41.817505 25035 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:25:41.816844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52684 (local address 127.2.56.61:44149)
0928 12:25:41.816877 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:41.816879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:41.817056 (+ 177us) server_negotiation.cc:407] Connection header received
0928 12:25:41.817082 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:41.817083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:41.817094 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:41.817145 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:41.817446 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52684: BlockingRecv error: recv got EOF from 127.0.0.1:52684 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 12:25:41.823063 25035 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:25:41.822252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52698 (local address 127.2.56.61:44149)
0928 12:25:41.822287 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:41.822289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:41.822506 (+ 217us) server_negotiation.cc:407] Connection header received
0928 12:25:41.822596 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:41.822597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:41.822607 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:41.822633 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:41.823005 (+ 372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52698: BlockingRecv error: recv got EOF from 127.0.0.1:52698 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 12:25:41.832926 25035 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:25:41.832219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52712 (local address 127.2.56.61:44149)
0928 12:25:41.832253 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:41.832255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:41.832464 (+ 209us) server_negotiation.cc:407] Connection header received
0928 12:25:41.832555 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:41.832556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:41.832566 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:41.832592 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:41.832884 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52712: BlockingRecv error: recv got EOF from 127.0.0.1:52712 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 12:25:41.851351 25035 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:25:41.850463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52720 (local address 127.2.56.61:44149)
0928 12:25:41.850518 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:41.850519 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:41.850781 (+ 262us) server_negotiation.cc:407] Connection header received
0928 12:25:41.850882 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:41.850883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:41.850894 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:41.850924 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:41.851294 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52720: BlockingRecv error: recv got EOF from 127.0.0.1:52720 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260928 12:25:41.885648 25035 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:25:41.884705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52736 (local address 127.2.56.61:44149)
0928 12:25:41.884759 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:41.884761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:41.885039 (+ 278us) server_negotiation.cc:407] Connection header received
0928 12:25:41.885143 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:41.885144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:41.885156 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:41.885182 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:41.885588 (+ 406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52736: BlockingRecv error: recv got EOF from 127.0.0.1:52736 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 12:25:41.951653 25035 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:25:41.951025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52738 (local address 127.2.56.61:44149)
0928 12:25:41.951080 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:41.951083 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:41.951177 (+ 94us) server_negotiation.cc:407] Connection header received
0928 12:25:41.951248 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:41.951249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:41.951259 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:41.951300 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:41.951599 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52738: BlockingRecv error: recv got EOF from 127.0.0.1:52738 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260928 12:25:42.081609 25035 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:25:42.080920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52750 (local address 127.2.56.61:44149)
0928 12:25:42.080984 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:42.080986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:42.081057 (+ 71us) server_negotiation.cc:407] Connection header received
0928 12:25:42.081124 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:42.081125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:42.081137 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:42.081184 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:42.081547 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52750: BlockingRecv error: recv got EOF from 127.0.0.1:52750 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 12:25:42.181639 25126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 713f2698a564411394b8eaacd670a85b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "713f2698a564411394b8eaacd670a85b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "713f2698a564411394b8eaacd670a85b" member_type: VOTER last_known_addr { host: "127.2.56.62" port: 34089 } } peers { permanent_uuid: "3419fc1d6f8b499c96017e7529f4dbd2" member_type: VOTER last_known_addr { host: "127.2.56.61" port: 44149 } } peers { permanent_uuid: "34c516a6495c436a9ec4dd2083f14e62" member_type: VOTER last_known_addr { host: "127.2.56.60" port: 38015 } } }
I20260928 12:25:42.181800 25126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 713f2698a564411394b8eaacd670a85b [sys.catalog]: This master's current role is: LEADER
W20260928 12:25:42.339274 25035 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:25:42.338673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52762 (local address 127.2.56.61:44149)
0928 12:25:42.338731 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:42.338733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:42.338744 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:25:42.338830 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:42.338831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:42.338843 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:42.338888 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:42.339210 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52762: BlockingRecv error: recv got EOF from 127.0.0.1:52762 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 12:25:42.709546 25123 catalog_manager.cc:1259] Loaded cluster ID: 6fbdf63fb42e4044a360d97916680aff
I20260928 12:25:42.709618 25123 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 34c516a6495c436a9ec4dd2083f14e62: loading cluster ID for follower catalog manager: success
I20260928 12:25:42.710421 25123 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 34c516a6495c436a9ec4dd2083f14e62: acquiring CA information for follower catalog manager: success
I20260928 12:25:42.710697 25123 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 34c516a6495c436a9ec4dd2083f14e62: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 12:25:42.796082 25171 catalog_manager.cc:1259] Loaded cluster ID: 6fbdf63fb42e4044a360d97916680aff
I20260928 12:25:42.796183 25171 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3419fc1d6f8b499c96017e7529f4dbd2: loading cluster ID for follower catalog manager: success
I20260928 12:25:42.797119 25171 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3419fc1d6f8b499c96017e7529f4dbd2: acquiring CA information for follower catalog manager: success
I20260928 12:25:42.797529 25171 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3419fc1d6f8b499c96017e7529f4dbd2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 12:25:42.855607 25240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:42.852445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52776 (local address 127.2.56.61:44149)
0928 12:25:42.852574 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:42.852576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:42.852640 (+ 64us) server_negotiation.cc:407] Connection header received
0928 12:25:42.852702 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:42.852703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:42.852759 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:42.852862 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:25:42.853067 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:42.853632 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:42.854602 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:42.854813 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:42.854881 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:42.855126 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:25:42.855270 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:25:42.855382 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:42.855467 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 12:25:42.855492 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":60,"threads_started":1}
I20260928 12:25:42.860050 25246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:42.857584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55268 (local address 127.2.56.60:38015)
0928 12:25:42.857735 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:42.857737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:42.857748 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:25:42.857835 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:42.857835 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:42.857884 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:42.857968 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:25:42.858059 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:42.858581 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:42.859329 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:42.859563 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:42.859637 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:42.859748 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:25:42.859874 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:25:42.859932 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:42.859950 (+ 18us) server_negotiation.cc:299] Negotiation successful
0928 12:25:42.859970 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":66,"threads_started":1}
I20260928 12:25:42.864735 25246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:42.861756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55284 (local address 127.2.56.60:38015)
0928 12:25:42.861820 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:42.861822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:42.861985 (+ 163us) server_negotiation.cc:407] Connection header received
0928 12:25:42.862084 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:42.862086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:42.862123 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:42.862153 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:25:42.862337 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:42.862849 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:42.863994 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:42.864206 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:42.864235 (+ 29us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:42.864457 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:25:42.864499 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:25:42.864558 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:42.864620 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 12:25:42.864649 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 12:25:42.864771 25240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:42.861672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52790 (local address 127.2.56.61:44149)
0928 12:25:42.861724 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:42.861726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:42.861950 (+ 224us) server_negotiation.cc:407] Connection header received
0928 12:25:42.861989 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:42.861990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:42.862005 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:42.862053 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:25:42.862172 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:42.862918 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:42.863978 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:42.864269 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:42.864328 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:42.864502 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:25:42.864544 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:25:42.864589 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:42.864669 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 12:25:42.864701 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 12:25:42.865084 25252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:42.861700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50030 (local address 127.2.56.62:34089)
0928 12:25:42.861968 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:42.861970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:42.861981 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:25:42.862034 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:42.862035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:42.862146 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:42.862236 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:25:42.862389 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:42.863123 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:42.863939 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:42.864253 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:42.864328 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:42.864584 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:25:42.864704 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:25:42.864759 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:42.864949 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 12:25:42.864997 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":53,"threads_started":1}
I20260928 12:25:42.866711 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 24888
I20260928 12:25:42.872390 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 24963
I20260928 12:25:42.877877 2272 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkXRk4O/build/release/bin/kudu with pid 25038
2026-09-28T12:25:42Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2671 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5395 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (270242 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST